arXiv CSopen access

Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation Jiashu Yao1 , Heyan Huang1 , Zeming Liu2 , Yuhang Guo1 * 1 Beijing Institute of Technology 2 Beihang University

arXiv:2604.11611v1 [cs.CL] 13 Apr 2026

Abstract To overcome the sparse reward challenge in reinforcement learning (RL) for agents based on large language models (LLMs), we propose Mutual Information Self-Evaluation (MISE), an RL paradigm that utilizes hindsight generative self-evaluation as dense reward signals while simultaneously calibrating them against the environmental feedbacks. Empirically, MISE enables an agent to learn autonomously from dense internal rewards supplementing sparse extrinsic signals. Theoretically, our work provides the first formal foundation for the paradigm of generative self-rewarding. We prove that utilizing hindsight self-evaluation rewards is equivalent to minimizing an objective that combines mutual information with a KL divergence term between the policy and a proxy reward policy. This theoretical insight then informs and justifies our calibration step, which actively aligns these rewards with the optimal policy. Extensive experiments show that MISE outperforms strong baselines, enabling open-source LLMs about 7B parameters to achieve performance comparable to GPT-4o on validation without expert supervision.

Figure 1: An LLM agent ( ) is tasked with interacting with a environment ( ) to complete a task ( ), getting intermediate rewards accumulated to a final score ( ). The environmental rewards are sparse, i.e., most environmental intermediate rewards are zero, so many strategically valuable actions are not properly rewarded.

seen tasks through interaction with the environments without supervision (Voss and Jovanovic, 2023; Morris et al., 2024), is the issue of reward sparsity (Ladosz et al., 2022; Deng et al., 2024). As exemplified in Figure 1, in an agent task, only a few actions directly leading to partial completion of the task (e.g. chop tomato with knife) receive non-zero rewards, while many strategically valuable actions (e.g., examine cookbook and take knife) are not properly rewarded. As a result, in reinforcement learning without step-wise labels, LLMs may suffer from low sample efficiency. Although traditional RL methods have been widely explored to address reward sparsity, few of them can be easily applied to LLM-based agents, as discussed in Section 2. To address the challenge of sparse rewards in autonomous adaptation in the era of LLMs, one simple approach is to follow the sucess of genera-

Introduction

Text agent tasks present sequential decision making challenges, employing natural language to describe the environments and the actions (Osborne et al., 2022). Leveraging large language models (LLMs) (Brown et al., 2020; Touvron et al., 2023; Achiam et al., 2023) agents provides an effective simulation of real-world agent-environment interaction. Recent researches (Song et al., 2024; Deng et al., 2024; Zeng et al., 2025) have been exploring finetuning LLM agents through reinforcement learning (RL), demonstrating the their potential to learn and adapt in simulated worlds. One important problem to overcome before achieving agents’ autonomously adaptation to un* Corresponding authors.

Methods

AI-Feedback Label-Free Dense-Reward Reinforce Calibrate

SFT PPO (Schulman et al., 2017) RFT (Yuan et al., 2023) DPO (Rafailov et al., 2024) ETO (Song et al., 2024) StepAgent (Deng et al., 2024) GRM (Mahan et al., 2024) PRM (Xi et al., 2025)

% % % % % " " "

% " " % % % " %

% % % % % " % "

% " % " " " " "

% % % % % " % %

MISE (ours)

Table 1: Comparison of MISE with the relevant methods. MISE leverages self-evaluation feedback rather than expert supervision to generate informative dense rewards. Meanwhile, MISE calibrates the self-evaluation to keep the dense rewards consistent with environmental rewards, avoiding overfitting to potential evaluation bias.

tive reward models (GRMs) (Ryu et al., 2024; Mahan et al., 2024; Yuan et al., 2024; Wu et al., 2024) and process reward models (PRMs) (Choudhury, 2025; Xi et al., 2025) in reasoning tasks, leveraging the generalized language understanding capabilities inherent in LLMs for step-wise action utility evaluations. Specifically, in agent tasks, an LLM can evaluate the strategic value of its own generated actions at each step, providing a valuable internal reward signals to augment sparse extrinsic rewards.

grates sparse environmental rewards with dense internal rewards derived from self-evaluation, while simultaneously applying calibration to avoid potential biases in the self-evaluation process rewards. Experiments shows that MISE can significantly improve the performance of LLMs agents, making open-sourced models sizing ∼7B exceed GPT-4omini and match GPT-4o on valid test. Furthermore, analysis shows that both action selection and self-evaluation components achieve performance improvement, realizing mutual assistance and enhancement within our proposed MISE framework. Our contributions can be summarized as follows:

However, in our concerned LLM agent tasks, such generative process reward models exhibit significant biases if they are not specifically trained with additional data. A specific example detail in Section 5.3 shows that, current models tend to overvalue the inventory action (81% cases judged as positive), encouraging the agent to repeatedly check the items currently held rather than making realistic progress. When trained with such biased vanilla PRM, the inventory action constitutes a disproportionately high 24.28% of all generated actions, leading to significant performance drop.

• We provide a theoretical establishment for hindsight self-evaluation process rewards, demonstrating that using self-evaluation as a reward is related to minimizing the mutual information between the actions and the future trajectory. The theoretical finding then informs and justifies the RL paradigm designs. • For the autonomous adaptation of LLM agents, we propose MISE, an RL paradigm that introduces self-evaluations as dense rewards in addition to sparse environmental rewards, while applying calibration to avoid overfitting to significant evaluation biases.

In this work, we fill the gap between the promise of PRMs-assisted LLM agent reinforcement and the significant biases of self-evaluation. We firstly conduct theoretical analysis, and establish the interpretation of hindsight self-evaluation by relating it to minimizing the mutual information (MI) between an agent’s action and its future trajectory. The theoretical discussion not only offers deep understanding of the PRMs, but also informs and justifies the RL paradigm design. Empirically, we introduce Mutual Information Self-Evaluation (MISE), a novel reinforcement learning paradigm that inte-

• Extensive experiments and analysis validate the high effectiveness of MISE, and prove the mutual assistance and enhancement effects between agents’ action selection and selfevaluation. 2

Related work

2.3

2.1

Adaptation for LLM Agents

Using LLMs as alternatives to rule-based or expertbased evaluators has been emerging due to their remarkable language capability and cost effectiveness. This trend has led to the emergence of generative reward models (GRMs) (Ryu et al., 2024; Mahan et al., 2024; Yuan et al., 2024; Wu et al., 2024), which is to train a model to align with the internal rewards of another model (Lee et al., 2024; Sun et al., 2024; Ahn et al., 2024; Li et al., 2024) or itself (Ryu et al., 2024; Cao et al., 2024). However, current LLM evaluators still struggle with issues of unreliability and lack of robustness (Gu et al., 2024). In the context of LLM agent, there still exists a lack of theoretical grounding and significant vulnerability to evaluation biases (Xi et al., 2025; Choudhury, 2025). Other than applying hindsight self-evaluation in LLM agent tasks for the first time, our approach overcomes two shortcoming in existing literature. First, it proposes a robust RL calibration paradigm to prevent overfitting to potential evaluation biases. Second, it mitigates the theoretical unreliability of LLM evaluation by proving its equivalence to minimizing the mutual information term and an inter-policy KL term.

LLM agents are tasked with sequential decisionmaking problems that use natural languages to convey environment observations and agent actions (Osborne et al., 2022). These tasks provide a efficient way for agents to learn reasoning, science, morality, etc. (Wang et al., 2022; Shi et al., 2022). We focus on task-agnostic free-form agents, where no priori components such as admissible command lists or task-specific knowledge graphs (Adhikari et al., 2020; Murugesan et al., 2021) are involved. In this flexible and challenging scenario, however, current adaptation methods either rely on step-wise supervision or fail to provide dense rewards, as shown in Table 1. Compared to previous methods, our proposed MISE does not rely on additional annotations. Instead, MISE generates dense informative rewards for efficient reinforcement exclusively using the LLM agent alone. Importantly, MISE applies calibration to self-evaluation to keep it consistent with external feedback, avoiding overfitting to the significant biases. 2.2

Reward Sparsity

It has been a major problem in RL that sequential decision making often associates with sparse rewards (Ladosz et al., 2022), where only a few actions leading to partial completion of the tasks get non-zero rewards. There has been many successful methods to address reward sparsity before LLM emerges, including curiosity-driven learning (Pathak et al., 2017; Burda et al., 2018), auxiliary task optimizing (Jaderberg et al., 2016; Riedmiller et al., 2018), and hindsight experience replay (Andrychowicz et al., 2017). However, previous methods are not suitable for our concerned tasks, as they either requires priori knowledge or construction specific to the task, or do not fit the LLM RL algorithm (e.g., PPO). Although StepAgent (Deng et al., 2024) proposes to address reward sparsity in LLM agent tasks, it has compulsory requirements for expert annotations. MISE can also be viewed as an reward sparsity mitigation approach. Its novelty lies in that utilizing and calibrating the superior general language understanding of LLMs for self-evaluation in text agent tasks for the first time, in addition to action selection. This intuitively simple approach is not only empirically effective but theoretically sound.

LLM-based Rewarding

Method

We firstly formulate hindsight process rewarding in details, then offer a theoretical interpretation of it, and finally propose MISE based on previous discussion. 3.1

Hindsight Process Reward

Let S, A, and O denote state, action and observation space respectively, given a sub-trajectory τ<t =< o0 , a0 , ..., ot−1 , at−1 , ot >,

(1)

where o ∈ O and a ∈ A are previous observations and actions, a policy πθ sample an action at ∈ A at ∼ πθ (·|τ<t ).

(2)

The environmental reward rE (st , at ) can evaluate the progress at each step, where st ∈ S is the state after conducting τ<t . However, rE is usually very sparse, i.e., most rewards are zero. In these cases, it can be difficult for an agent to efficiently learn from the environment. To address sparsity in rE , we propose hindsight process reward rP , which assigns a non-zero reward to each action given the whole (including the 3

future) trajectory. Collectively, the optimization target is defined as J(θ) = Eτ ∼πθ

T h i X rE (st , at )+rP (τ<t , at , τ>t ) . t=1

(3) Note that in Equation 3, the process reward rP is always dependent on the future trajectory τ>t when evaluating at . 3.2

Figure 2: Optimizing towards hindsight self-evaluation ideally enables the agent (πθ in the top) to make decisions as effective as those made with full future sight (πP in the bottom).

Theoretical Discussion

Here, we offer a theoretical discussion on the hindsight process reward rP and show its relation with mutual information. We firstly construct a proxy policy πP to represent the process reward rP with a Gibbs distribution πref (at |τ<t ) rP πP (at |τ<t , τ>t ) = , exp Z(τ<t , τ>t ) β (4) where Z is the normalization term, πref is the reference policy, and β is the temperature parameter. We can hereby prove that minimizing the KL divergence between the original and proxy policy is equivalent to maximizing the KL regularized process reward, as detailed in Appendix A. Further deriving the KL divergence, we have

those made with full knowledge of the future (πP ). As is shown in the example of Figure 2, an LLMbased policy πθ may not be able to choose the right action in the first place. However, optimizing toward rP pulls its distribution closer to the proxy policy πP , which can see the entire trajectory and find the utility of the action take knife. KL divergence Different from the mutual information term, the KL divergence term is to let policy πθ mimic a marginalized distribution πP (a|τ<t ), without introducing effective future information such as τ>t . Since πP (a|τ<t ) is not explicitly calibrated during vanilla RL training, the KL divergence term may lead to biases in the optimization target. The KL divergence term informs the need of actively calibrating self-evaluations for a reliable process rewarding.

3.3

Mutual Information Self-Evaluation (MISE)

Based on the discussion above, we propose mutual information self-evaluation (MISE), an RL paradigm that utilizes the mutual information to address sparsity, and calibrates the process rewards to avoid self-evaluation biases. As is shown in Figure 1, in additional to the sparse environmental reward rE , MISE adds two rewards to the vanilla RL process, namely, the reward of hindsight process self-evaluation rP : τ<t × a × τ>t → {−1, 1} and the calibration reward rC : τ × rc → [−1, 1]. Hindsight process rewards rP directly addresses the reward sparsity problem, as it uses the general language understanding ability of LLMs to assign a non-zero score to each action given the whole trajectory. Specifically, the description of the whole trajectory is given in the prompt as shown in Appendix B, and an LLM is instructed to evaluate

Mutual information Minimizing mutual information implies that the agent can select optimal actions in the present without explicitly searching the entire future trajectory. Ideally, this allows the agent (πθ ) to make choices that are as effective as 4

Experiments

4.1

Test bench

Task Our main experiments and analyses are conducted on First TextWorld Problems (FTWP) (Inc., 2019). This task is built on TextWorld (Côté et al., 2018), with the aim of building an autonomous agent that can navigate and interact within a textsimulated modern house and finally cook a meal. For generalizability, we conduct experiments on other agent tasks including ScienceWorld (Wang et al., 2022) and WebShop (Yao et al., 2022). Figure 3: MISE framework overview. Apart from environmental sparse rewards (rE ), MISE generates dense process self-evaluation rewards (rP ), which are calibrated by rewards (rC ) to be consistent with actual performance. MISE adapts LLM agents with dense rewards without overfitting to self-evaluation biases.

the strategic value of each action within the context of the entire trajectory to assign either −1 or 1.

Figure 4: The overall experimental procedure. Each individual steps can be included or skipped.

Calibration rewards rC , on the other hand, regularizes the consistency between the self-evaluation and the real performance, to prevent LLMs from overfitting the potentially biased rewards. We design heuristic scores as rC ,

Procedure The experimental procedure is illustrated in Figure 4. The process consists of three stages. First, and optionally, LLMs can learn through imitation of expert demonstrations. SFT methods and those requiring data annotations fall within this initial stage. Second, LLMs learn solely through interaction with the environment in a completely label-free manner. This second stage embodies the concept of autonomy of adaptability and is the context in which our proposed MISE operates. Third, LLMs are evaluated on unseen environments to assess the generalizability.

rC = clip(1.2 − 2 × |preal − pself-eval |, 0, 1) × 2 − 1, (6) where given a trajectory τ and the maximum environmental score S, preal ∈ [0, 1] and pself-eval ∈ [0, 1] are defined as PT

t=1 rE (st , at )

preal =

, S t=1 rP (τ<t , at , τ>t ) . T

PT pself-eval =

External rewards The external rewards refer to the scores provided by environments indicating the completion of a sub-goal, which are sparse as most are zeros, neglecting many strategically valuable actions. The step j-th reward for action rEj is formulated as the average of the suffix scores:

(7)

Generally, rC is aimed to make the selfevaluation of LLMs on a trajectory consistent to its real performance. A detailed discussion and justification of the rC design is in Appendix C.

T X rEj = −1 + 2 × [ si / (T − j + 1)],

(9)

i=j

Algorithm Combining Equation 3 with the two proposed rewards rP and rC , the optimization target of MISE is

where si denotes the environmental score of step i. Metrics The performance of LLMs are evaluated with micro success rate (Suc Rate). Suc Rate indicates the degree of task completion, i.e., the rate at which the agent achieves the given goal. We

T hX i rE + rP J(θ) = E + rC − βDKL . (8) 2 τ ∼πθ t=1

Average

Models

FTWP

ScienceWorld

WebShop

FF Rate Suc (V) Suc (T) Suc (V) Suc (T) Suc (V) Suc (T) Suc (V) Suc (T) GPT-4o GPT-4o-mini

82.13 72.54

40.40 32.55

38.29 29.64

43.66 27.26

51.08 31.63

40.65 38.60

32.41 31.98

36.88 31.79

31.39 25.30

LLaMA3-8B + ReAct + RFT + online DPO + PPO + PRM + MISE

67.83 68.68 77.47 67.47 78.90 80.70 77.57

28.92 28.30 31.29 29.54 28.20 32.75 36.48

23.06 23.63 26.10 24.40 22.33 27.36 30.47

21.73 18.81 24.75 22.54 27.87 28.45 34.71

17.24 17.80 22.32 20.09 24.48 25.61 28.90

36.12 37.97 38.64 36.76 26.04 38.87 43.40

31.79 33.10 34.02 32.14 20.03 33.65 38.18

28.90 28.13 30.48 29.31 30.69 30.94 31.34

20.14 19.98 21.97 20.96 22.47 22.81 24.34

Qwen2-7B + PPO + PRM + MISE

44.85 71.34 70.92 73.14

19.60 27.86 28.28 32.12

15.72 22.15 22.95 26.22

16.20 30.69 31.80 33.80

13.11 20.85 22.81 24.90

12.35 22.09 22.60 28.87

10.67 21.85 22.24 28.61

30.26 30.80 30.43 33.68

23.38 23.75 23.81 25.15

Gemma2-9B + PPO + PRM + MISE

61.23 86.42 88.90 89.74

9.93 25.87 26.63 28.07

9.74 20.67 21.45 23.25

16.30 36.95 37.77 40.14

17.30 29.83 30.50 32.94

13.49 40.66 42.12 44.08

11.91 32.19 33.85 36.80

0.0 0.0 0.0 0.0

Table 2: Main experiments on three tasks. Bolded numbers denote the best results for each base model.

also report the format faithfulness rate (FF Rate) (Yao et al., 2024), which refers to the percentage of commands that can be compiled and executed by the environment, representing the degree to which LLMs can obey the given command format. 4.2

directly optimizes the LLMs for higher environmental rewards. (5) PRM (Mahan et al., 2024) uses LLM self-evaluation as additional rewards based on vanilla PPO. We further experiment whether annotation-free MISE can take effects in scenarios with trajectory labels by post-tuning adapted models, comparing with methods requiring labels, including: (1) SFT, (2) DPO-based methods like ETO (Song et al., 2024) and StepAgent-Implicit (Deng et al., 2024).

Experimental setup

The key experimental setup are listed here, and more details are described in Appendix D and E. Models We conduct experiments on LLaMA38B-Instruct, Qwen2-7B-Instruct, and Gemma2-9BInstruct. We mainly conduct comparison experiments on LLaMA3, as its original model performs best. We also compare MISE with close-sourced models including GPT-4o and GPT-4o-mini.

Hyper-parameters We employ commonly used hyperparameters without cherry-picking to ensure the reliability of the experiments. In all LLM adaptations, we use LoRA (Hu et al., 2022) with rank being 64. SFT uses a linear scheduled learning rate 3e − 6, trains for 1 epoch with batch size 256. DPO uses a linear scheduled learning rate 3e − 7, trains for 1 epoch with batch size 32. PPO uses a constant learning rate 1e − 5, trains for 3 epochs with batch size being 32, KL coefficient (β in Equation 8) being 0.1. All inferences use greedy decoding.

Baselines We compare our autonomous adaptation method MISE to other strong label-free baselines. (1) ReAct (Yao et al., 2023) encourages LLMs to think before act using prompting techniques. (2) RFT (Yuan et al., 2023) collects the successful sampled trajectories to construct a dataset used for SFT LLMs. (3) Online DPO (Rafailov et al., 2024) optimizes LLMs by training on pairwise comparisons of preference data, which is generated by the models online. (4) PPO (Schulman et al., 2017), the de-facto standard for RL in LLMs,

4.3

Results

The results in the label-free scenario on FTWP, ScienceWorld, and WebShop tasks are shown in Table 2. Without any external labels, MISE significantly 6

improves the task completion rate, outperforming previous methods like RFT, online DPO, PPO, and PRM by a large margin, matching or even exceeding GPT-4o and GPT-4o-mini. Notably, the superiority of MISE over PRM validates the effectiveness of calibrating self-evaluations with the calibration rewards (rC ). In the experimental results, we have two outlier observations, including (1) PPO performance degradation with LLaMA3 on ScienceWorld, and (2) less improvement of all methods on WebShop compared to on ScienceWorld. The first observation aligns with findings from previous work (Song et al., 2024). For the second observation, we speculate this is because the average trajectory length in WebShop (∼ 5) is significantly shorter than in the other two tasks (∼ 19), making process optimization more challenging. As a notable specific case, both the original and RL-finetuned Gemma models fail to follow the online shopping format instructions, resulting in no success on WebShop. Models

5.1

43.66 27.26

51.08 31.63

86.46 84.80 84.66 84.79

66.54 67.04 67.21 68.45

58.53 59.81 59.69 62.66

Table 3: Experiment results on FTWP task in the scenario with training labels. All adaptations are based on LLaMA3-8B. Additional details for SFT are listed in Appendix E.

Although MISE is an label-free RL method aiming for autonomy of adaptability, it can also improves the performance of SFT-tuned models through post-tuning. As shown in Table 3, MISE based on a SFT model outperforms both its reference model and other RL-based methods, and surpassing strong close-sourced models such as GPT-4o. A discussion on how MISE can help SFT models improve is shown in Appendix G.

Quality of self-evaluation

MISE is to use self-evaluation as dense internal rewards to improve action selection, while simultaneously calibrate the self-evaluation process by optimizing the consistency between self-evaluation and the actual task completion rate. Ideally, after training with MISE, LLMs will excel at both action generation and self-evaluation. We firstly analyze the quality of the LLMs’ self-evaluations. In doing so, we first construct a evaluation dataset. we randomly select 44 instances which contains 764 actions, and manually label each action to determine whether it is strategically useful for the ultimate goal. Then we test LLMs on the datasets with two metrics, (1) extractability, i.e., whether the generated evaluation can pull out a meaningful score, and (2) accuracy, i.e., whether the extracted evaluation is the same as human-given labels. The results are shown in Table 4. It is evident that with the rewards for self-evaluation rC , MISE can improve the evaluation capability in the context of LLM agents. Meanwhile, without those rewards punishing self-evaluations for being inconsistent with actual results, it can suffer from a decline due to catastrophic forgetting (Luo et al., 2023). Moreover, from the perspective of probably approximately correct (PAC) learnability (Valiant, 1984; Angluin and Laird, 1988), the existence of noise in evaluations or labels necessitates larger training datasets. Given that the self-evaluations produced by LLaMA3-based agents are not as accurate as those from larger models such as GPT-4o, there is significant room for improvement. While the performance margin between MISE and previous methods is already substantial, it has a great potential to obtain even better performance when employing more powerful base models.

FF Rate Suc(Val) Suc (Test)

GPT-4o 82.13 GPT-4o-mini 72.54 SFT ETO StepAgent MISE

ine the mutual assistance and enhancement effects in LLM agents between the action selection and the self-evaluation components: (1) Are step-wise evaluations generated by LLMs of high quality? (2) Does the self-evaluation help improve LLMs’ action selection? (3) How does the calibration for process rewarding (rC ) regularization affects agent action selection?

Analasis

Since our proposed MISE method centers on leveraging self-evaluation to improve autonomy of adaptability, we are particularly interested in a deeper understanding of this evaluation process. In this section, we present three analyses to exam-

5.2

Impact of self-evaluation on acting

Having examined the quality of self-evaluation, we now turn to another crucial question: to what extent 7

Models

Self-Evaluation Quality

Task Completion Quality

Extractability

Accuracy

FF Rate

Suc Rate (Val)

Suc Rate (Test)

GPT-4o

97.73

85.22

82.13

43.66

51.08

LLaMA3 LLaMA3 + PPO LLaMA3 + MISE + w/o rm (vanilla GRM) + random eval + flipped eval

79.55 75.00 86.36 68.18 65.45 2.49

67.28 68.13 72.33 65.91 64.20 57.89

67.83 78.90 77.57 80.86 77.84 2.81

21.73 27.87 34.71 28.45 23.69 0.47

17.24 24.48 28.90 25.61 19.03 0.24

Table 4: Analytic experiments with optional corruption of self-evaluation process. Results of both self-evaluation and task completion are assessed. Note that MISE without the reward for self-evaluation rC is equivalent to the vanilla PRM approach. Best results based on LLaMA3 models are shown in bold.

does the self-evaluation quality influences LLM action selection efficacy? To investigate this, we progressively corrupt the self-evaluations at three increasing levels of severity. (1) Training without rewards for selfevaluation: as demonstrated in Section 5.1, this scenario results in a slight degradation of evaluation accuracy. (2) Training with random self-evaluations: this scenario introduces noise into the training process. (3) Training with flipped self-evaluations: this scenario represents a more severe corruption, where the evaluator deliberately provides false evaluations. The results are presented in Table 4. With the first level of corruption (training without rewards for self-evaluation), the model experiences a performance decrease, yet still outperform PPO, as even the corrupted self-evaluations retain some useful dense information. At the second level (training with random self-evaluations), the model’s performance falls below that of PPO due to the introduced random noise, but still surpasses the original LLaMA3 as the environment continues to provide some sparse feedback. At the third level of corruption (flipped self-evaluations), the model is unable to learn effectively, instead generating nonsensical outputs that attempt to conform to the deliberately misleading evaluations. 5.3

Nevertheless, as demonstrated in the preceding two subsections, rC exhibits a significant positive influence on both self-evaluation quality and action selection performance. To better understand the crucial role of rC in the MISE training process, we examine the trainingtime task completion rate and the positive selfevaluations rate epoch by epoch, comparing models trained with rC (our proposed MISE) to those without it (vanilla PRM).

Figure 5: The task completion rates and positive selfevaluation rates in three epochs on the valid set. MISE w/o denotes MISE without self-evaluation calibration rewards.

The results are illustrated in Figure 5. Without rC , the LLMs exhibit overconfidence in their action selections. Although, during training, MISE without rC lags behind MISE with rC in terms of actual task completion rates, it assigns itself significantly higher self-evaluations (up to 60% positive). The underlying reason for the overconfidence is the inherent significant bias of LLM selfevaluations. Self-evaluation is not a perfect reflection of the real action plausibility within given environments. Instead, it can favor certain actions that are not actually useful for the completion of the task. While self-evaluation effectively addresses

Rewards for self-evaluation

MISE is composed of three rewards, namely environmental rewards (rE ), self-evaluation process rewards (rP ), and self-evaluation calibration rewards (rC ). Among these, rC is unique in that it does not directly optimize the action selection process. Instead, it focuses on calibrating the self-evaluation. 8

reward sparsity, it as a part of optimization target can mislead the agent to generate actions that are perceived as good by itself but detrimental to performance. On the other hand, the self-evaluation calibration rewards (rC ) can help avoid the overfitting to potential biases by rcalibrating the selfevaluation to be consistent with the real completion rate. As a result, MISE can simultaneously mitigate reward sparsity and self-evaluation biases. A concrete example of this bias is the tendency of LLaMA3-8B to overvalue the inventory action, which checks the items currently held by the agent. Without rm , the inventory action constitutes a disproportionately high 24.28% of all generated actions. However, with the inclusion of rm , this percentage decreases to 14.34%, only slightly higher than the 8.51% in the original model.

Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. 2024. Tuning large multimodal models for videos using reinforcement learning from ai feedback. arXiv preprint arXiv:2402.03746. Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. 2017. Hindsight experience replay. Advances in neural information processing systems, 30. Dana Angluin and Philip Laird. 1988. Learning from noisy examples. Machine learning, 2:343–370. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Conclusion Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. 2018. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355.

In this work, we present MISE, an RL paradigm for LLM autonomous agent adaptations. MISE encourages the agent to learn from its self-evaluation but not conform to the potential bias, by simultaneously utilizing and calibrating generative selfevaluation rewards, which is theoretically justified by its relation to minimizing the mutual information. Extensive experiments validate the high effectiveness of MISE with or without step-wise labels across diverse models and tasks. Moreover, analyses prove the mutual assistance and enhancement effects in LLM agents between the action selection and the self-evaluation components, suggesting the potential of self-evaluation in autonomous adaptation. MISE facilitates efficient and effective autonomous agent adaptation without intensive expert labeling, presenting a step towards autonomy of adaptability in the reinforcement of LLM agents.

Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng. 2024. Enhancing reinforcement learning with dense rewards from language model critic. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9119–9138. Sanjiban Choudhury. 2025. Process reward models for llm agents: Practical framework and directions. arXiv preprint arXiv:2502.10325. Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2018. Textworld: A learning environment for textbased games. CoRR, abs/1806.11532. Zhirui Deng, Zhicheng Dou, Yutao Zhu, Ji-Rong Wen, Ruibin Xiong, Mang Wang, and Weipeng Chen. 2024. From novice to expert: Llm agent policy optimization via step-wise reinforcement learning. arXiv preprint arXiv:2411.03817.

References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594.

Ashutosh Adhikari, Xingdi Yuan, Marc-Alexandre Côté, Mikuláš Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and Will Hamilton. 2020. Learning dynamic belief graphs to generalize on text-based games. Advances in Neural Information Processing Systems, 33:3045– 3057.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.

Microsoft Canada Inc. 2019. First textworld problems: A reinforcement and language learning challenge. https://competitions.codalab.org/competi tions/21557.

Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. 2017. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778–2787. PMLR.

Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. 2016. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. 2022. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1–22.

Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. 2018. Learning by playing solving sparse reward tasks from scratch. In International conference on machine learning, pages 4344–4353. PMLR.

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and 1 others. 2024. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In Forty-first International Conference on Machine Learning.

Sangwon Ryu, Heejin Do, Yunsu Kim, Gary Geunbae Lee, and Jungseul Ok. 2024. Multi-dimensional optimization for text summarization via reinforcement learning. arXiv preprint arXiv:2406.00303. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

Hao Li, Xue Yang, Zhaokai Wang, Xizhou Zhu, Jie Zhou, Yu Qiao, Xiaogang Wang, Hongsheng Li, Lewei Lu, and Jifeng Dai. 2024. Auto mc-reward: Automated dense reward design with large language models for minecraft. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16426–16435.

Zijing Shi, Meng Fang, Yunqiu Xu, Ling Chen, and Yali Du. 2022. Stay moral and explore: Learn to behave morally in text-based games. In The Eleventh International Conference on Learning Representations.

Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747.

Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and error: Exploration-based trajectory optimization of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7584–7600, Bangkok, Thailand. Association for Computational Linguistics.

Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, JanPhilipp Fränken, Chelsea Finn, and Alon Albalak. 2024. Generative reward models. arXiv preprint arXiv:2410.12832.

Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Daniel Cox, Yiming Yang, and Chuang Gan. 2024. Salmon: Selfalignment with instructable reward models. In The Twelfth International Conference on Learning Representations.

Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg. 2024. Position: Levels of agi for operationalizing progress on the path to agi. In Forty-first International Conference on Machine Learning.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Keerthiram Murugesan, Mattia Atzeni, Pavan Kapanipathi, Kartik Talamadupula, Mrinmaya Sachan, and Murray Campbell. 2021. Efficient text-based reinforcement learning by jointly leveraging state and commonsense graph representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 719–725.

Leslie G Valiant. 1984. A theory of the learnable. Communications of the ACM, 27(11):1134–1142. Peter Voss and Mladjan Jovanovic. 2023. Why we don’t have agi yet. arXiv preprint arXiv:2308.03598.

Philip Osborne, Heido Nomm, and Andre Freitas. 2022. A survey of text games for reinforcement learning informed by natural language. Transactions of the Association for Computational Linguistics, 10:873– 887.

Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. 2022. Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279–11298.

Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. 2024. Meta-rewarding language models: Self-improving alignment with llm-as-ameta-judge. arXiv preprint arXiv:2407.19594. Zhiheng Xi, Chenyang Liao, Guanyu Li, Yajie Yang, Wenxiang Chen, Zhihao Zhang, Binghai Wang, Senjie Jin, Yuhao Zhou, Jian Guan, and 1 others. 2025. Agentprm: Process reward models for llm agents via step-wise promise and progress. arXiv preprint arXiv:2511.08325. Jiashu Yao, Heyan Huang, Zeming Liu, Haoyu Wen, Wei Su, Boao Qian, and Yuhang Guo. 2024. Reff: Reinforcing format faithfulness in language models across varied tasks. arXiv preprint arXiv:2412.09173. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable realworld web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 3. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825. Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, Yang Katie Zhao, and Mingyi Hong. 2025. Reinforcing multi-turn reasoning in llm agents via turn-level credit assignment. In ICML 2025 Workshop on Computer Use Agents.

Construction of Proxy Policy

As a result, using adapted absolute differences between preal and pself-eval is able to pulling the optimization direction of rP towards the real environmental direction rE . Figure 7 (left) illustrates the function of rC with the absolute difference between the positive selfevaluation rate and the task completion rate as the independent variable. As depicted in the figure, rC penalizes significant inconsistencies between the two rates, while not requiring them to be perfectly identical, recognizing that some degree of difference is expected. Figure 7 (right) gives an example that how different self-evaluations and their corresponding rewards (rewards for self-evaluation) are rewarded when the actual task completion rate is 0.7.

Given the definition of πP πP (at |τ<t , τ>t ) = πref (at |τ<t ) rP (τ<t , at , τ>t ) , exp Z(τ<t , τ>t ) β

(10)

take the logarithm of both sides of the equation, log πP (at |τ<t , τ>t ) = rP (τ<t , at , τ>t ) log πref (at |τ<t ) + β − log Z(τ<t , τ>t ),

(11)

and substitute it into the KL expansion, we have DKL (π ∥ πP ) = Eτ ∼π [log π(at |τ<t ) − log πP (at |τ<t , τ>t )] h π(at |τ<t ) i = Eτ ∼π log πref (at |τ<t ) 1 − Eτ ∼π rP (τ<t , at , τ>t ) + log Z(τ<t , τ>t ) β 1 = DKL (π ∥ πref ) − Eτ ∼π rP (τ<t , at , τ>t ) β + log Z. (12) Since Z is not related to the policy to optimize (π), we have min DKL (π ∥ πP ) ⇐⇒ π h i max Eτ ∼π rP − βDKL (π ∥ πref ) .

TextWorld environments is able to provide a rich set of environmental information at each step, including reachable items, admissible commands, current inventory, and so on. However, to ensure the generality of the challenge, we provide the LLM-based agents with only the feedback received in response to their actions, and no other environmental information is supplied as input. Although we set a maximum of 20 steps for the agents, this could still lead to excessive memory consumption. To mitigate this, during both the generation and adaptation, we retain only the most recent 10 rounds of conversation. Furthermore, during adaptation, we may retain fewer rounds of history so that the prompt always contains fewer than 1280 tokens.

(13)

Prompts

The prompts used are shown in Figure 6.

Input Representations

Discussion on Heuristic Reward Design

The self-evaluation calibration reward (rC ) is designed to prevent self-evaluation from deviating significantly from actual environmental feedback, thereby preventing the LLM-based agent from being misled. Consequently, we constructed rC heuristically to encourage closer alignment between the positive self-evaluation rate and the actual task completion rate, as shown in Equation 6. Similar to the construction of the proxy policy shown in Appendix A, we can show that the optimal trajectory distributions under KL regularization for rE and rP can be defined respectively as

Implementation Details

E.1

Hardware and Framework

All model adaptation is performed using NVIDIA A800 80GB GPUs, while inference is run on NVIDIA TITAN RTX 24GB GPUs. All adaptation methods, including SFT-based methods (SFT, RFT), DPO-based methods (online DPO, ETO, StepAgent), and PPO-based methods (PPO, MISE), are implemented using the trl library1 . For trajectory-wise DPO (ETO), we developed a custom data collator that ignore all token labels except those belonging to assistant messages, thereby focusing the learning process exclusively on the assistant’s behaviors.

PE (τ ) ∝ exp(rE (τ )), PP (τ ) ∝ exp(rP (τ )). (14)

https://github.com/huggingface/trl

Figure 6: Prompts for agents (left) and self-evaluation (right).

Figure 7: The graph (left) and illustration (right) for reward for self-evaluation (rm ). rm punishes preal and pc for being inconsistent, while not requiring them to be perfectly identical.

E.2

pairs. Fine-tuning on the dataset takes less than 20 minutes on one NVIDIA A800.

Hyperparameters

The hyperparameters used in the experiments are listed in Table 5. E.3

Online DPO For each of the 222 valid environments, two trajectories are generated using the original model with a temperature setting of 1.0. A pairwise comparison is then performed between the two trajectories within each environment. If one trajectory has a superior score, both trajectories are added into the preference dataset. This process yielded a preference dataset containing 84

Data and Computational Cost for Label-Free Methods

RFT We take the hypothesis on valid set (222 environments) of the original model, and filter out all the trajectories yielding a score of zero, resulting in a dataset containing 1, 781 prompt-action 13

Hyper-Parameters

SFT-Based Methods

PPO-Based Methods

DPO-Based Methods

LoRA rank LoRA α LoRA modules LoRA dropout

64 64 all linear 0.0

learning rate lr scheduler warmup ratio epoch batch size

3e − 6 linear 0.1 1 256

1e − 5 constant 0.0 3 32

3e − 7 linear 0.1 1 32

0.1* -

0.3

PPO KL coefficient DPO β

Table 5: Hyperparameters for the experiments. SFT-based methods include SFT, RFT; PPO-based methods include PPO, MISE; DPO-based methods include online DPO, ETO, StepAgent. The asterisk symbol denotes that, we use KL coefficient of 0.1 with LLaMA3 and Gemma2. With Qwen2, we use higher coefficient of 0.2. This adjustment was made because Qwen2, being initially less capable, tends to diverge at an early training stage with the lower KL coefficient. Increasing the KL coefficient helps to stabilize the training process for the model.

StepAgent StepAgent is also based on SFT model. To collect StepAgent dataset, we firstly operate a teacher-forcing generation on training set, which is to collect the data of how the model acts differently to the reference label given the same history. The teacher-forcing generation on training set takes about 11 hours on one NVIDIA TITAN RTX. Then, the collected step-wise preference data is used for step-wise DPO, which costs about 32 hours.

trajectories. Subsequent trajectory-wise DPO utilizing this dataset costs less than 20 minutes. PPO and MISE These two methods directly use the 222 valid environments to conduct label-free reinforcement learning. Vanilla PPO takes about 28 hours, and MISE takes about 48 hours. For Mise on Gemma2, we additionally adopt 4-bit quantization to fit the model in to 80GB GPU memory, and the quantized RL process takes about only 22 hours. E.4

Data and Computational Cost for Label-Supervised Methods

SFT We directly use the walkthrough labels of the training set (4440 trajectories) to conduct SFT, which takes about 9 hours.

ScienceWorld and WebShop Settings

In our research, the primary experiments were conducted on FTWP (Inc., 2019) to investigate core hypotheses. Meanwhile, we conduct additional experiments to assess the generalizability of our findings across diverse agent tasks. Specifically, we adopt two datasets: ScienceWorld (Wang et al., 2022) and WebShop(Yao et al., 2022). The ScienceWorld dataset simulates an educational platform, encapsulates structured interactions between learners and scientific content, including problemsolving trajectories, conceptual queries, and engagement patterns across various STEM domains. On the other hand, the WebShop dataset simulates an e-commerce platforms, requiring an agent to navigate and purchase according to given demands and budgets. By incorporating these datasets, we aim to demonstrate that MISE transcends domainspecific constraints, effectively addressing agent tasks with different features.

SFT for MISE In addition to walkthrough labels, GPT-4o is employed to evaluate all labels, and these evaluations are incorporated into the SFT dataset to improve the self-evaluation capability of the model. The inclusion of this supplementary data does not significantly increase the training load. The SFT process still takes about 9 hours on one NVIDIA A800 GPU. ETO The process of ETO is quite similar to the trajectory-wise online DPO, with three differences: (1) ETO is based on the SFT model, (2) the data is collected based on training set instead of valid set, and (3) the preferred trajectory is derived from walkthrough labels rather than a superior generated hypothesis. ETO costs about 4 hours. 14

On the two additional tasks, all the preprocessing, training, and decoding details are the same to those with FTWP, except that (1) the maximum step on WebShop task is set to 10 instead of 20, as the environment feedback from WebShop environment is significantly longer than the others, (2) The KL coefficient of Qwen2 PPO training is 0.1 instead of 0.2, and (3) the environmental rewards in WebShop are "w/o IR" defined in Equation 9, as the intermediate rewards are not given in this environment. Moreover, regarding the evaluation of ScienceWorld and WebShop, all task instances within have the same maximum score, making the MicroRate and MacroRate identical, so we only report MicroRate here.

Insights into Improvement over Supervision

Experiments in Table 3 shows that even after supervised training on a large dataset, LLM-based agents can still benefit from label-free MISE reinforcement learning on a relatively small dataset. This is reasonable, as the introduction of MISE also introduces new data. Meanwhile, we offer a specific perspective on bias in training labels to explain this improvement. Unlike some tasks with one single correct solution, text-based games often allow for multiple winning trajectories. However, even among these correct trajectories, some may be considered suboptimal due to the inclusion of inefficient or redundant actions. A common flaw observed in our training labels is the consistent attempt to try to open a door when moving between rooms connected by a door, regardless of whether the door is already open. When applying SFT with these suboptimal labels, the fine-tuned model will exhibit flaw in its generated trajectories as well. In test set, the SFT-trained agent attempts to open already open doors for 909 times, a stark contrast to the original LLaMA3-8B model, which only does that for 15 times. However, when MISE is applied to the SFTtrained agent, the number of such attempts drops to 665. This reduction may occur because the selfevaluation mechanism can identify the error based on the environmental feedback indicating that the door is already open, and subsequently provide negative feedback for such actions, thus discouraging them.

Record · ID 10349 · SHA-256 17e44e25ac27451d
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.