arXiv CS open access

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning

2026 · arxiv_cs

arXiv CS · Papers · 2026

deep-learning machine-learning neural-networks

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning Haoyu Zheng1,3 * , Yun Zhu2 , Qing Wang3 , Wenqiao Zhang1† 1 Zhejiang University 2 Shanghai AI Laboratory 3 Tencent

arXiv:2608.07371v1 [cs.LG] 7 Aug 2026

Abstract

should be distributed among them. Existing work reduces this granularity mismatch through finer rollout or step-level groupings (Wang et al., 2025; Feng et al., 2025), but remains centered on estimating outcome-based advantages. Hindsight distillation complements sparse outcome rewards with dense supervision by reevaluating realized actions with their turn-aligned realized consequences (Agarwal et al., 2024; Zhao et al., 2026). Recent methods use hindsight-induced policy discrepancies to modulate token updates, distribute supervision within steps, or select turns for feedback (Yang et al., 2026; Lu et al., 2026a; Li et al., 2026b; Zhang et al., 2026; Yeo et al., 2026). However, a completed rollout can produce discrepancies at many turns. Each signed token gap encodes a local revision, but its raw magnitude does not state how strongly that turn should be emphasized relative to other turns in the same interaction. The resulting problem is how to preserve signed token-level revision while redistributing hindsight supervision across turns under a calibrated allocation profile. We introduce TRIAL,1 a trajectory-relative hindsight distillation framework with one shared protocol across interactive environments. For every decision turn, an outcome view exposes the locally realized consequence of that decision to a training-time scorer, which reevaluates the same realized response. The resulting signed token-level log-probability gap determines update direction and local strength. To allocate this supervision across turns, TRIAL aggregates the absolute gaps within each turn and compares the turn’s share of total revision magnitude with its share of eligible tokens. This produces a trajectory-relative profile whose turn multipliers have an eligible-tokenweighted mean of one, emphasizing turns with

Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards. However, a completed rollout can yield many such signals, leaving their appropriate allocation across turns unclear. We introduce TRIAL, a trajectory-relative hindsight distillation framework with a unified turn-aligned scoring protocol. For each decision turn, TRIAL extracts an outcome view of that decision’s realized consequence and evaluates the same response under ordinary and hindsight-conditioned contexts. The signed log-probability gap determines the direction and local strength of token-level supervision, while turn-level magnitudes are normalized jointly over the realized trajectory. The resulting allocation multipliers have an eligibletoken-weighted mean of one, redistributing dense supervision across turns while fixing its average multiplier. Experiments on WebShop and ALFWorld with different backbones show that TRIAL outperforms GRPO across all eight combinations of backbone, environment, and evaluation metric, while achieving the best or tied-best performance among six methods on six of them. On WebShop with Qwen3-1.7B, TRIAL improves the success rate from 56.4% to 75.2% and the task score from 78.7% to 85.7%. Controlled ablations further show that trajectory-relative turn allocation provides substantial gains beyond those of dense hindsight distillation alone.

Introduction

Agentic reinforcement learning commonly optimizes sparse outcome rewards. GRPO, for example, compares complete rollouts for the same task using relative outcome signals (Shao et al., 2024). A multi-turn rollout, however, contains many decisions made under changing observations, and its final outcome does not specify how supervision

* Work done during internship at Tencent. †

Code is available at Chihaya-Anon-chan/TRIAL.

Corresponding author.

https://github.com/

Across rollouts

compare complete interactions

τ₁

a₁

o₂

a₂

o₃

τ₂

a₁

o₂

a₂

o₃

τ₃

a₁

o₂

a₂

o₃

cases. On WebShop with Qwen3-1.7B, it improves the success rate from 56.4% to 75.2% and the task score from 78.7% to 85.7%. Controlled ablations keep the hindsight pathway fixed while comparing the trajectory-relative profile with unit and sourceshuffled profiles. The trajectory-relative profile outperforms the unit profile on all four WebShop and ALFWorld metrics, showing that relative turn allocation provides benefits beyond dense hindsight distillation alone. Our contributions are:

rolloutrelative outcome one signal per rollout

Within one completed rollout realized action–observation consequences

a₁

o₂

ordinary policy view

a₂

o₃

same actions

a₃

o₄

hindsight policy view

• We formulate trajectory-relative hindsight allocation as a joint calibration problem that couples signed token-level revision with relative turn emphasis while controlling the average allocation multiplier.

policy-view comparison

trajectory-relative signed revision

outcome-relative trajectory-relative

ONE POLICY UPDATE

hindsight is training-only

• We introduce TRIAL, which derives signed token supervision from ordinary and hindsight-conditioned contexts and constructs a trajectory-normalized turn profile. Its eligible-token-weighted mean is one, so redistribution across turns keeps the average multiplier fixed.

Figure 1: Complementary feedback granularities in a multi-turn agent. Outcome feedback compares complete trajectories. At each decision turn, a turn-aligned outcome view exposes the realized action–observation consequence (a → o) in a hindsight-conditioned context. Comparing the ordinary and hindsight contexts on the same realized tokens yields signed token-level supervision, whose magnitude is normalized across turns to produce a trajectory-relative allocation profile.

• We evaluate TRIAL against five baselines on two interactive environments with two backbone models. Full-set comparisons and controlled ablations demonstrate consistent improvements over GRPO and benefits beyond dense hindsight distillation alone.

stronger revision evidence per token while keeping the average allocation multiplier fixed. The resulting profile reweights the signed token gaps to form a dense hindsight objective. GRPO continues to provide rollout-level outcome supervision, while TRIAL allocates hindsight supervision among tokens and turns within each rollout. The profile normalization keeps the average allocation weight fixed, and a scalar clamp bounds the dense contribution relative to the detached GRPO loss magnitude without changing GRPO’s advantage construction. All hindsight contexts, discrepancy scores, and allocation profiles are used only during training, so deployment retains the ordinary online policy without additional inference cost. Figure 1 summarizes this decomposition. We evaluate TRIAL on WebShop (Yao et al., 2022) and ALFWorld (Shridhar et al., 2021) using Qwen2.5-3B and Qwen3-1.7B. TRIAL outperforms GRPO across all eight combinations of backbone, environment, and evaluation metric, and ranks best or tied best among six methods in six

Related Work

Outcome-relative optimization for language agents. Instruction-following models have progressed from fine-grained visual grounding and referring (Yuan et al., 2024, 2025, 2026) to architectures that adapt a shared backbone across heterogeneous multimodal tasks and domains (Zhang et al., 2024; Lin et al., 2025), and are increasingly deployed as agents that plan and act over many turns (Zheng et al., 2026b; Gao et al., 2026). Training such agents from environment outcomes is dominated by group-relative objectives. GRPO estimates a critic-free advantage by comparing rewards within a rollout group (Shao et al., 2024), but sparse multi-turn returns obscure which decisions matter. This mismatch compounds as each action changes later observations while reward often arrives only at termination. RAGEN analyzes these dynamics (Wang et al., 2025); GiGPO adds state-aligned step groups (Feng et al., 2025); and concurrent sibling 2

3.1

or counterfactual methods shape token advantages (Ding et al., 2026; Meng and Chen, 2026). These methods refine outcome-relative credit: local structure changes how reward-directed preference is estimated. TRIAL leaves that comparison unchanged and instead allocates hindsight-induced policy revision among turns after interaction.

For a task x, the rollout policy πold samples a group of G multi-turn trajectories. A trajectory τ = (o1 , a1 , o2 , . . . , aK , oK+1 ) contains each response ak = (yk,1 , . . . , yk,Tk ) together with its preceding observation ok and realized post-action observation ok+1 . Given the final environment return Ri of rollout i, GRPO computes

On-policy hindsight distillation and fine-grained allocation. Hindsight Experience Replay relabels realized experience as additional goals (Andrychowicz et al., 2017), while asymmetric actor–critic uses training-only information without changing deployed policy inputs (Pinto et al., 2018). On-policy language-model distillation, including multi-turn variants, learns from studentgenerated sequences (Agarwal et al., 2024; Hübotter et al., 2026; Wang et al., 2026a; Zheng et al., 2026a). Our direct baselines use this signal differently: OPSD scores a privileged same-model view, RLSD and SDAR couple distillation with reward learning, and SERL selectively distills multi-turn hindsight (Zhao et al., 2026; Yang et al., 2026; Lu et al., 2026a; Li et al., 2026b). None of these baselines supplies a mean-one, trajectory-relative turn profile. TCOD progressively expands the trajectory depth exposed to the student, while HiSR uses hindsight likelihood ratios to modulate segmentlevel process rewards (Wang et al., 2026b; Lu et al., 2026b). Other methods reweight tokens or select particular steps and turns (Xu et al., 2026; Li et al., 2026a; Zhang et al., 2026; Yeo et al., 2026). Concurrent work explores turn-aware schedules and selection (Tan et al., 2026; Zhou et al., 2026a,b). TRIAL instead couples all eligible turns in a completed trajectory: same-token hindsight discrepancy defines revision mass, eligible-token mass provides its reference, and their ratio yields a meanone profile that retains signed local revision while calibrating relative emphasis across turns.

Preliminaries

bi = Ri − R , A σR + ϵ A

(1)

where R and σR are the mean and standard deviation within the rollout group. GRPO then optimizes its standard clipped surrogate loss LGRPO (Schulman et al., 2017; Shao et al., 2024). TRIAL leaves this advantage and the GRPO clipping rule unchanged. At token t of turn k, hk,t denotes the ordinary interaction history, including the response prefix yk,<t . A binary mask mk,t marks tokens eligible P k for hindsight learning. We use nk = Tt=1 m for PK k,t the eligible tokens in turn k and Nτ = k=1 nk for the trajectory total. In our experiments, all generated response tokens are eligible, including reasoning tokens when they are present in the generated response; prompt and padding tokens are excluded. 3.2

Trajectory-Relative Hindsight Allocation

Hindsight-conditioned policy gap. Once a realized trajectory is available, the same hindsight protocol is applied at every decision turn. It constructs the turn-aligned outcome view zk = OutcomeView(τ, k),

(2)

which exposes the locally realized consequence of the decision at turn k. An environment interface maps native interaction records into this shared turn-aligned object. In our experiments, the frozen training-time scoring snapshot is the pre-update rollout snapshot, πT = πold , so the profile-building gap ∆old isolates context augmentation. The scoring snapshot is a training-time policy snapshot rather than a separately trained teacher, and the realized response being scored is unchanged. Conceptually, the two token contexts are

Method

TRIAL complements GRPO’s comparison across rollouts with a relative hindsight update within each rollout. The completed interaction provides a training-only policy view that reveals how the probability of each realized token changes under hindsight. Each signed gap supplies token-level update direction and local revision strength. TRIAL aggregates the absolute gaps into a trajectory-normalized profile that reallocates this dense signal across turns. Figure 2 summarizes the resulting update.

hk,t = [Pk ; yk,<t ], Pk+ = Aug(Pk , zk ),

(3)

+ h+ k,t = [Pk ; yk,<t ],

where Pk is the ordinary prompt at turn k. The realized trajectory provides one outcome view for 3

Shared rollout outcome signal Â2

Rollout-relative outcome feedback

group norm

Â2

signed gaps

Â1

t10

≈

+ + + + +

Â3

t11

realized postaction evidence: a 2 → o3

Turn-weighted signed gaps t6

hindsightaugmented context

t14

Â2 outcome + hindsight

ordinary context

t13

Trajectory-relative hindsight reallocation

t12

+ -

Policy Update

≈

Across-turn allocation by gap magnitudes normalized within τᵢ → wₖ

Figure 2: TRIAL augments GRPO on the same realized trajectory. GRPO’s across-rollout comparison yields one bi shared by all action tokens in rollout i. A frozen training-time scoring view evaluates realized outcome signal A tokens with hindsight derived from action–observation consequences; signed gaps determine token direction and local strength, while a normalized turn profile determines relative allocation. The aligned signals form one token update field, and deployment retains only the ordinary policy.

the turn; the recorded response and its token prefix are otherwise unchanged. Appendix A.3 details the shared protocol and its serialization. For any policy π evaluated on the ordinary history, let clipb (u) clip u to [−b, b] and define " ∆k,t (π) = sg clipc log

πT (yk,t | h+ k,t ) π(yk,t | hk,t )

trajectory average removes the common gap scale. The remaining ratio is the TRIAL turn weight: 1 X sk sτ = nj sj , wk = , Nτ sτ j:nj >0 (6) X nk wk = Nτ .

k:nk >0

The last identity gives the weights an eligible-tokenweighted mean of one. Equivalently, wk compares turn k’s share of the absolute hindsight gap with its share of eligible tokens. A weight above one therefore identifies a turn whose revision is larger than its token count alone would predict. If the total gap is numerically zero, TRIAL uses wk = 1; a single eligible turn also reduces to unit weighting. The profile is invariant to a common positive rescaling of all token gaps in the trajectory. This normalization makes the allocation comparable to a unit-profile control: both expose the eligible tokens to a mean multiplier of one, but TRIAL redistributes that multiplier across turns. The score sk measures the magnitude of a hindsightconditioned policy discrepancy, not the causal importance of turn k to the final reward. For example, if one turn changes substantially under hindsight while two equally long turns barely change, TRIAL concentrates the dense update on the first turn without changing the trajectory’s mean multiplier. For three equally long turns with s = (0.1, 0.2, 0.3),

(4) where sg denotes stop-gradient. Here clipc is an elementwise bound on the log-probability gap, while the outer clamp in Eq. 8 bounds the scalar dense contribution. We write ∆old k,t = ∆k,t (πold ) when building the allocation profile and ∆θk,t = ∆k,t (πθ ) during actor optimization. Equation 4 follows onpolicy self-distillation: the frozen scoring view has access to hindsight, while the target remains the student’s own sample (Agarwal et al., 2024; Zhao et al., 2026). Turn score and relative allocation. For every turn containing at least one eligible token, TRIAL summarizes the absolute hindsight gap: T

sk =

k 1 X mk,t ∆old k,t . nk

(5)

t=1

Turns with nk = 0 are omitted from the profile and receive no dense update. The token-weighted 4

for example, sτ = 0.2 and w = (0.5, 1, 1.5). The mean-one identity calibrates allocation rather than loss magnitude: signed token gaps and token losses still determine the realized dense update. 3.3

interaction records into the same turn-aligned semantic view. The semantic scoring, trajectory normalization, and optimization rule are shared across environments; implementation schedules and resource settings are specified in Appendix A.

Joint Optimization Algorithm 1 TRIAL training update

Dense signed hindsight objective. During actor optimization, TRIAL refreshes the detached θ gap P ∆τ,k,t against the current policy. Let M = τ,k,t mτ,k,t be the number of eligible tokens in the actor microbatch. The dense hindsight objective is the masked token mean Ldense = −

1 X mτ,k,t wτ,k ∆θτ,k,t M τ,k,t

Input: actor πθ , task batch {xb }, interaction interface (OutcomeView, Aug). 1. Freeze the pre-update snapshot πold and use it as the scoring snapshot πT while collecting groups of completed on-policy trajectories {(τi , Ri )}. bi and the outcome objective Lout from group 2. Compute A returns. 3. For each turn k, extract zi,k = OutcomeView(τi , k) and + + form Pi,k = Aug(Pi,k , zi,k ) and h+ i,k,t = [Pi,k ; yi,k,<t ]. 4. Score the same realized tokens under ordinary and augmented contexts; compute ∆old i,k,t , si,k , and the detached trajectory-normalized profile wi,k . 5. During actor optimization, refresh ∆θi,k,t , construct Ldense , and update θ with Eq. 8. 6. Discard the hindsight evidence, augmented contexts, and profile. Output: updated ordinary policy πθ .

(7)

× log πθ (yτ,k,t | hτ,k,t ). We set this loss to zero when the microbatch contains no eligible tokens. A positive gap increases the probability of the realized token, whereas a negative gap decreases it; its absolute value sets the local strength before turn reweighting. Thus, Eq. 6 controls where the dense update is concentrated, while Eq. 4 preserves its signed token-level direction and local strength.

Deployment uses only πθ with the ordinary interaction history and requires no hindsight module, teacher, or routing procedure.

Joint objective. Let Lout denote the sparse outcome-driven policy objective, including LGRPO and any fixed regularization. At update g, TRIAL optimizes bg = α sg(|LGRPO |) , L(θ) = Lout (θ) + clipbg (λg Ldense ) .

Experiments

4.1

Experimental Setup

Benchmarks, models, and evaluation. We evaluate Qwen2.5-3B-Instruct (Qwen et al., 2024) and Qwen3-1.7B (Yang et al., 2025) on WebShop (Yao et al., 2022) and ALFWorld (Shridhar et al., 2021). Training uses online rollouts from each environment’s official training split; the task batch supplies scheduled task instances, while interaction histories, rewards, and hindsight signals are obtained from the live environment interaction. WebShop reports success and dense task score on the complete official test split of 500 goals. ALFWorld reports success on every game in the official Seen and Unseen evaluation splits (140 and 134 games, respectively), keeping the two splits separate. Trainingtime validation uses a fixed 128-instance diagnostic subset and is distinct from the final evaluation procedure. Every reported method is finally run with the same complete-set evaluator and benchmark metric implementation.

(8)

The coefficient λg is zero during warmup and equals the configured dense-feedback coefficient afterward. Applied once to the actor-microbatch scalar, the clamp bounds the hindsight contribution relative to the detached GRPO loss magnitude. When the dense term saturates either boundary, its gradient is zero and the outcome-policy gradient remains active. The allocation profile is constructed from the frozen scoring snapshot and detached before actor optimization. The actor therefore receives a fixed trajectory-level allocation target while refreshing only the signed token gap in the current policy; this separates where supervision is assigned from the policy update that applies it. Algorithm 1 summarizes the shared optimization procedure. The interaction interface implements only OutcomeView and Aug, translating native

Baselines. The comparison set includes samestack GRPO (Shao et al., 2024) and implementations of SERL (Li et al., 2026b), SDAR (Lu et al., 2026a), GRPO+OPSD (Zhao et al., 2026), 5

and RLSD (Yang et al., 2026). We preserve the released methods’ native objectives and recommended settings rather than rewriting them as variants of TRIAL.

method completes more goals while also improving partial task completion. ALFWorld complements this outcome comparison with longer action sequences and separate seen and unseen evaluations. On Qwen3-1.7B, TRIAL exceeds the strongest competing average by 3.6 points on Seen and 6.0 points on Unseen. The category breakdown sharpens this result: different baselines lead different task families, whereas TRIAL attains the strongest overall average on both splits without relying on a single dominant category. With Qwen2.5-3B, it raises the Seen and Unseen averages over GRPO by 20.7 and 22.4 points, respectively, and ties the strongest Seen average. We use Qwen3-1.7B ALFWorld for the main-body learning-dynamics analysis and both environments for the controlled profile study. WebShop’s paired metrics show that the aggregate gain is not confined to marginal goal completion. ALFWorld’s family columns likewise show that the split-level averages are not driven by one task type. Together, the benchmarks evaluate exact and partial task completion alongside seen and unseen household interaction rather than repeating one aggregate comparison.

Unified hindsight protocol. Both environments use the protocol in Eq. 2: each decision turn is paired with a training-only outcome view of its locally realized consequence, and the same recorded response is scored with and without that view. The interaction interface maps native event records into this shared turn-aligned view; the same-response scoring rule, trajectory-level normalization, and deployment interface are shared. The outcome-view pathway is held fixed among Uniform, Permuted, and TRIAL within each controlled comparison, so these experiments vary the allocation profile rather than the hindsight construction. Controlled-study configuration. The profile study compares Uniform, Permuted, and TRIAL with Qwen3-1.7B on both environments; GRPO provides the outcome-only reference. Within each environment, the three hindsight configurations share batch, rollout, trajectory-length, and optimization settings. All systems train for 150 optimization steps, and group-relative methods sample eight trajectories per task instance. Main-body dynamics use Qwen3-1.7B ALFWorld, whose longer trajectories make turn allocation easiest to inspect. The appendix specifies the hindsight serialization, full hyperparameters, and hardware. 4.2

4.3

Controlled Revision Profiles

The controlled study compares GRPO, Uniform, Permuted, and the complete method with Qwen31.7B on both WebShop and ALFWorld (Table 2). Uniform, Permuted, and TRIAL hold the dense hindsight pathway fixed and vary only the applied turn profile; GRPO is the outcome-only reference. Uniform uses unit multipliers. Permuted assigns each active turn another active turn’s multiplier within the same trajectory, preserves relative multiplier ratios up to a common rescaling, and restores a token-weighted mean of one. Uniform provides the unit-weighted control for dense hindsight feedback. Relative to GRPO, it improves WebShop success by 6.4 points but does not yield consistent gains on the other three metrics. Unit-weighted dense hindsight therefore does not explain the full gain. Applying TRIAL’s gapdependent, source-aligned relative profile raises all four metrics over Uniform: WebShop success and score increase by 12.4 and 7.9 points, and ALFWorld Seen and Unseen success increase by 8.6 and 11.2 points. Permuted provides a stricter correspondence control. It preserves the non-uniform multiplier values up to normalization but assigns them to differ-

Overall Agentic Performance

Table 1 reports the complete comparison in one matrix, including all six ALFWorld task families rather than only split-level averages. Across the eight aggregate environment metrics (ALFWorld Seen and Unseen averages plus the two WebShop metrics for each backbone), TRIAL is higher than GRPO in every case and records or matches the highest point estimate among all six methods in six. The pattern is strongest with Qwen3-1.7B, where TRIAL leads both WebShop metrics and both ALFWorld split averages. With Qwen2.5-3B, TRIAL also gives the highest WebShop success, ties the highest ALFWorld Seen average, and improves all four aggregate metrics over GRPO. The strongest gain appears with Qwen3-1.7B on WebShop. Relative to GRPO, TRIAL increases success from 56.4% to 75.2% and dense task score from 78.7% to 85.7%. The two metrics agree: the 6

ALFWorld success Method

WebShop

Seen

Unseen

Succ. Score

P&P Two Look Heat Cool Clean Avg. P&P Two Look Heat Cool Clean Avg. Qwen2.5-3B GRPO 80.0 41.7 SERL 97.1 66.7 SDAR 91.4 50.0 GRPO+OPSD 94.3 54.2 RLSD 94.3 54.2 TRIAL 97.1 58.3

61.5 50.0 53.8 81.2 53.8 75.0 61.5 56.2 69.2 75.0 69.2 62.5

44.0 76.0 84.0 84.0 72.0 80.0

63.0 58.6 66.7 29.4 81.5 79.3 79.2 82.4 88.9 77.1 75.0 76.5 77.8 75.0 83.3 58.8 81.5 76.4 75.0 52.9 88.9 79.3 79.2 52.9

55.6 52.2 33.3 69.6 61.1 73.9 61.1 65.2 55.6 91.3 77.8 69.6

52.4 66.7 85.7 61.9 71.4 76.2

45.2 50.7 77.4 69.4 71.0 73.9 80.6 70.1 71.0 70.9 77.4 73.1

66.2 68.6 69.0 68.4 68.8 69.4

81.8 82.2 82.6 81.4 81.6 82.4

Qwen3-1.7B GRPO 80.0 33.3 SERL 71.4 45.8 SDAR 74.3 45.8 GRPO+OPSD 68.6 37.5 RLSD 71.4 41.7 TRIAL 77.1 41.7

61.5 68.8 46.2 68.8 69.2 62.5 53.8 56.2 53.8 56.2 61.5 62.5

64.0 60.0 60.0 64.0 56.0 68.0

63.0 62.9 70.8 23.5 70.4 62.1 62.5 47.1 74.1 65.0 75.0 35.3 63.0 58.6 58.3 41.2 74.1 60.7 62.5 47.1 88.9 68.6 66.7 64.7

50.0 69.6 55.6 56.5 44.4 69.6 55.6 60.9 44.4 52.2 50.0 82.6

61.9 57.1 66.7 52.4 47.6 61.9

61.3 58.2 61.3 57.5 71.0 62.7 54.8 54.5 74.2 56.7 77.4 68.7

56.4 67.6 70.0 60.8 68.4 75.2

78.7 81.1 81.8 79.1 81.4 85.7

Table 1: Complete-set results (%). ALFWorld is decomposed by task family; Avg. is split-level success rather than an unweighted family mean. WebShop reports success (Succ.) and task score. P&P denotes pick-and-place; Two denotes placing two objects. Bold and underlining mark the best and second-best result per backbone and column, respectively. (a)

(b)

TRIAL

60 40 20 0

warmup 25

Profile dispersion (std.)

GRPO

(c)

0.25

Mean turn score

Validation success (%)

0.23

0.21

0.19 50

100

Optimization step

125

150

0.30

0.26

0.22 25

100

Optimization step

125

150

100

Optimization step

125

150

Figure 3: Qwen3-1.7B validation and allocation dynamics on ALFWorld. (a) Success on the fixed 128-game validation set, evaluated every five optimization steps. Along this diagnostic training run, TRIAL exceeds GRPO at 22 of the 26 checkpoints after the dense term activates, and finishes at 70.3% versus 62.5%. (b–c) Mean absolute-gap turn score and standard deviation of the normalized turn profile, shown as non-overlapping five-step means on independent vertical scales. Shading marks steps 1–24 before the dense hindsight term activates; the dashed line marks step 25.

4.4

ent source turns. Its effect varies by environment, remaining above Uniform on ALFWorld while falling below it on WebShop. TRIAL nevertheless exceeds Permuted on all four aggregate metrics, by 18.4 and 9.4 points on WebShop and by 2.2 and 4.5 points on ALFWorld. The controls therefore distinguish dense hindsight, a non-uniform profile, and source-turn correspondence. TRIAL’s consistent lead supports applying revision to the turn whose policy assessment changes, rather than merely adding dense feedback or non-uniform weights.

Learning and Allocation Dynamics

Learning dynamics. We use Qwen3-1.7B ALFWorld as the representative long-horizon setting. The three panels report success on the fixed validation set, the mean absolute-gap turn score, and dispersion of the constructed trajectory-relative turnweight profile. Validation is evaluated every five optimization steps; the two mechanism diagnostics are non-overlapping five-step means. After the dense hindsight objective activates, TRIAL exceeds GRPO at 22 of the 26 diagnostic checkpoints along this run and ends 7.8 percentage points higher. The 7

ALFWorld success Method

WebShop

Seen

Unseen

Succ. Score

P&P Two Look Heat Cool Clean Avg. P&P Two Look Heat Cool Clean Avg. 61.5 68.8 38.5 62.5 53.8 75.0 61.5 62.5

GRPO 80.0 33.3 Uniform 71.4 45.8 Permuted 82.9 33.3 TRIAL 77.1 41.7

64.0 48.0 68.0 68.0

63.0 62.9 70.8 23.5 77.8 60.0 75.0 41.2 74.1 66.4 79.2 58.8 88.9 68.6 66.7 64.7

50.0 69.6 33.3 60.9 33.3 65.2 50.0 82.6

61.9 61.9 71.4 61.9

61.3 58.2 61.3 57.5 67.7 64.2 77.4 68.7

56.4 62.8 56.8 75.2

78.7 77.8 76.3 85.7

Table 2: Controlled revision-profile results with Qwen3-1.7B (%). ALFWorld Avg. is split-level success; WebShop reports success (Succ.) and task score. P&P denotes pick-and-place and Two placing two objects; bold marks each column’s best result. The three hindsight variants share the outcome-view protocol and optimization: Uniform uses unit multipliers, Permuted shuffles source-turn assignments and restores a token-weighted mean of one, and TRIAL uses the source-aligned relative profile. GRPO is the outcome-only reference.

Relative turn multiplier

(a)

First object

Recovery / search

(b)

Second object T14

2.0

Acquisition

T15 acquire

1.5

Placement T18

Uniform

1.0

Navigation T16

0.5 1

Turn in the complete successful trajectory

TURN

RECORDED ACTION

T14

INFORMATIVE DISCOVERY

go to countertop 1

countertop 1: ... → On peppershaker 2 ...

UPWEIGHTED

T16

INEFFECTIVE ACTION

→ Nothing happens.

DOWNWEIGHTED

T18

TASK COMPLETION

move the peppershaker 2 to → You the shelf 1.

take peppershaker 1 from countertop 1 move peppershaker 2 to shelf 1

NEXT OBSERVATION

ALLOCATION

Inspection/other Completion 0.9

1.0

1.1

1.2

Mean relative multiplier

1.3

NEAR MEAN

Figure 4: TRIAL’s hindsight allocation on successful ALFWorld Pick-Two trajectories. (a) An 18-turn profile and three example decisions; T15 is the acquisition following T14’s discovery. (b) Mean multipliers by decision class across 21 successful trajectories from five task instances. Acquisition is largest and completion remains near unit; error bars show SEM over trajectory-level class means.

warmup boundary is marked explicitly.

the trajectory mean, including attempted acquisitions that produce no state change. The pattern is not dominated by terminal turns: acquisition has the largest mean multiplier across 21 successful trajectories, while completion remains near the unit profile. TRIAL thus concentrates hindsightderived supervision where the turn-aligned outcome view most changes the policy’s assessment of the recorded decision, rather than treating every turn alike. Together, the complete-set results establish outcome-level gains, the profile controls test source-aligned allocation, and the diagnostics trace the same mechanism through training and realized trajectories without additional task-specific reward shaping or inference-time components.

Profile evolution. The diagnostic panels expose a complementary pattern. The mean turn score decreases from roughly 0.23–0.24 early in training to 0.19 at the end, while the standard deviation of the normalized profile rises from roughly 0.22– 0.23 to 0.30. In other words, the absolute hindsight gap contracts as training progresses, yet its relative distribution across turns becomes more heterogeneous. This joint trend matches TRIAL’s distinction between local gap strength and relative allocation across turns. Trajectory-level allocation. Figure 4 makes this heterogeneity concrete. In the illustrative Pick-Two success, TRIAL emphasizes both object-location discovery and object-acquisition decisions. Several ineffective or redundant interactions fall below 8

Conclusion

Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. 2026. Reinforcement learning via self-distillation. Preprint, arXiv:2601.20802.

TRIAL combines rollout-level outcome comparison with turn-aligned hindsight: signed local revisions form a mean-one trajectory-relative profile that reallocates supervision while preserving outcome advantages and the average multiplier. Across two environments and backbones, gains over GRPO, controls, and diagnostics support emphasizing decisions most revised by the outcome view.

Sijia Li, Yuchen Huang, Zifan Liu, Yanping Li, Jingjing Fu, Li Zhao, Jiang Bian, Ling Zhang, Jun Zhang, and Rui Wang. 2026a. GEAR: Granularity-adaptive advantage reweighting for LLM agents via selfdistillation. Preprint, arXiv:2605.11853.

Limitations

Xiaozhe Li, Tianyi Lyu, Yang Li, Yichuan Ma, Peiji Li, Linyang Li, Qipeng Guo, Dahua Lin, and Kai Chen. 2026b. What and when to distill: Selective hindsight distillation for multi-turn agents. Preprint, arXiv:2605.19447.

Our evidence currently covers text-based interactive environments with discrete actions. TRIAL also assumes that a completed trajectory exposes serializable post-action evidence that can be aligned with the decision that produced it. Extending this interface to less structured interaction remains future work. The hindsight view is confined to training; deployment retains the ordinary policy interface without an additional model, hindsight input, or routing procedure. Each update also requires a hindsight-conditioned forward pass over generated responses, increasing training cost relative to outcome-only GRPO. Our reported runs use a single seed; cross-seed variance and statistical robustness remain unmeasured.

Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li, Xiaohui Song, Siliang Tang, Jun Xiao, Hui Lin, Yueting Zhuang, and Beng Chin Ooi. 2025. HealthGPT: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation. In Proceedings of the 42nd International Conference on Machine Learning. Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. 2026a. Self-distilled agentic reinforcement learning. Preprint, arXiv:2605.15155. Zhicong Lu, Zichuan Lin, Wei Jia, Changyuan Tian, Deheng Ye, Peiguang Li, Li Jin, Nayu Liu, Guangluan Xu, and Wei Feng. 2026b. HISR: Hindsight information modulated segmental process rewards for multi-turn agentic reinforcement learning. Preprint, arXiv:2603.18683.

References Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. 2024. On-policy distillation of language models: Learning from self-generated mistakes. Preprint, arXiv:2306.13649.

Zibin Meng and Kani Chen. 2026. CRAFT: Counterfactual credit assignment from free sibling rollouts for self-distilled agentic reinforcement learning. Preprint, arXiv:2606.29476.

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. 2017. Hindsight experience replay. In Advances in Neural Information Processing Systems.

Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. 2018. Asymmetric actor critic for image-based robot learning. In Robotics: Science and Systems XIV.

Tianyu Ding, Jianhong Xin, and Juan Pablo De la Cruz Weinstein. 2026. Keep policy gradient in charge: Sibling-guided credit distillation for longhorizon tool-use agents. Preprint, arXiv:2606.12634.

Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2024. Qwen2.5 technical report. Preprint, arXiv:2412.15115.

Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. 2025. Group-in-group policy optimization for LLM agent training. In Advances in Neural Information Processing Systems. Mingjian Gao, Wenqiao Zhang, Yuqian Yuan, Yang Dai, Binhe Yu, Zheqi Lv, Haoyu Zheng, Jiaqi Zhu, Zhiqi Ge, Zixuan Wan, Siliang Tang, and Yueting Zhuang. 2026. VisualThink-VLA: Visual intermediate reasoning for effective and low-latency vision-languageaction policies. Preprint, arXiv:2605.30011.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.

Yuqian Yuan, Wentong Li, Zhaocheng Li, Yutong Lin, Juncheng Li, Siliang Tang, Jun Xiao, Yueting Zhuang, and Wenqiao Zhang. 2026. InstructSAM: Segment any instance with any instructions. Preprint, arXiv:2605.26102. Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, Jianke Zhu, and Lidong Bing. 2024. VideoRefer suite: Advancing spatial-temporal object understanding with video LLM. Preprint, arXiv:2501.00599.

Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. ALFWorld: Aligning text and embodied environments for interactive learning. Preprint, arXiv:2010.03768. Qitai Tan, Zefang Zong, Mo Li, Yipeng Shi, Yang Li, and Peng Chen. 2026. ATOD: Annealed turnaware on-policy distillation for multi-turn agentic tasks. Preprint, arXiv:2606.27814.

Yuqian Yuan, Wenqiao Zhang, Xin Li, Shihao Wang, Kehan Li, Wentong Li, Jun Xiao, Lei Zhang, and Beng Chin Ooi. 2025. PixelRefer: A unified framework for spatio-temporal object referring with arbitrary granularity. Preprint, arXiv:2510.23603.

Hao Wang, Guozhi Wang, Han Xiao, Yufeng Zhou, Yue Pan, Jichao Wang, Ke Xu, Yafei Wen, Xiaohu Ruan, Xiaoxin Chen, and Honggang Qi. 2026a. Skill-SD: Skill-conditioned self-distillation for multi-turn LLM agents. Preprint, arXiv:2604.10674.

Wenqiao Zhang, Tianwei Lin, Jiang Liu, Fangxun Shu, Haoyuan Li, Lei Zhang, He Wanggui, Hao Zhou, Zheqi Lv, Hao Jiang, Juncheng Li, Siliang Tang, and Yueting Zhuang. 2024. HyperLLaVA: Dynamic visual and language expert tuning for multimodal large language models. Preprint, arXiv:2403.13447.

Jiaqi Wang, Wenhao Zhang, Weijie Shi, Yaliang Li, and James Cheng. 2026b. TCOD: Exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents. Preprint, arXiv:2604.24005.

Yanfei Zhang, Xu Lin, and Chenglin Wu. 2026. StepOPSD: Step-aware online preference distillation for agent reinforcement learning. Preprint, arXiv:2605.27140.

Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. 2025. RAGEN: Understanding self-evolution in LLM agents via multi-turn reinforcement learning. Preprint, arXiv:2504.20073.

Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. 2026. Self-distilled reasoner: On-policy self-distillation for large language models. Preprint, arXiv:2601.18734. Haoyu Zheng, Yun Zhu, Shu Yuan, Shangming Chen, Qing Wang, Wenqiao Zhang, Jun Xiao, and Yueting Zhuang. 2026a. MAIGO: Mitigating lost-inconversation with history-cleaned on-policy selfdistillation. Preprint, arXiv:2605.27186.

Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. 2026. TIP: Token importance in on-policy distillation. Preprint, arXiv:2604.14084.

Haoyu Zheng, Yun Zhu, Yuqian Yuan, Bo Yuan, Wenqiao Zhang, Siliang Tang, and Jun Xiao. 2026b. PILOT: Planning via internalized latent optimization trajectories for large language models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics.

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.

Yuhang Zhou, Lizhu Zhang, Yifan Wu, Mingyi Wang, Bo Peng, Jiayi Liu, Xiangjun Fan, and Zhuokai Zhao. 2026a. SAGE-OPD: Selective agent-guided intervention for multi-turn on-policy distillation. Preprint, arXiv:2606.19659.

Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. 2026. Self-distilled RLVR. Preprint, arXiv:2604.03128.

Yuhang Zhou, Kai Zheng, Haoling Li, Dengyun Peng, Can Xu, and Jingjing Chen. 2026b. TurnOPD: Making on-policy distillation turn-aware for efficient longhorizon agent training. Preprint, arXiv:2607.05804.

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. WebShop: Towards scalable realworld web interaction with grounded language agents. In Advances in Neural Information Processing Systems, volume 35, pages 20744–20757. Woongyeng Yeo, Yumin Choi, Taekyung Ki, and Sung Ju Hwang. 2026. HINT-SD: Targeted hindsight self-distillation for long-horizon agents. Preprint, arXiv:2605.17873.

Experimental Protocol

Configuration Training task batch Rollouts per task PPO minibatch Microbatch per GPU Maximum prompt tokens Maximum response tokens Auxiliary coefficient Gap clipping bound c Auxiliary activation step Relative clamp α

This appendix records the protocol needed to interpret and reproduce the reported comparisons. It follows the order of the experimental argument: environment and metric definitions, optimization settings, construction of the training-only hindsight view, baseline implementation, and additional result breakdowns. A.1

Environments and Metrics

ALFWorld

16 8 64 8 4096 512 0.005 2.0 25 1.0

16 8 256 32 2048 512 0.01 2.0 25 1.0

Table 3: Training and TRIAL configurations. The learning rate is 1 × 10−6 and training lasts 150 optimization steps in both environments.

WebShop. WebShop (Yao et al., 2022) presents a shopping instruction and requires the agent to search, navigate product pages, choose options, and purchase an item. We cap interaction at 15 environment turns. Success is the percentage of goals completed exactly; task score gives partial credit for satisfying the requested attributes. Final training rollouts sample from the released training goal pool, while final results use the complete official test split of 500 evaluation goals.

25 onward. The auxiliary loss operates on generated response tokens. We clip each detached logprobability gap to [−2, 2] before constructing turn scores and set the scalar auxiliary-loss bound to the magnitude of the detached GRPO loss (α = 1). The reported main TRIAL runs and controlled profile studies set the frozen scoring snapshot to the pre-update rollout policy, πT = πold . Within each controlled comparison, Uniform, Permuted, and TRIAL therefore use the same scoring parameters; the compared intervention is the applied turn profile. Thus, ∆old compares the same parameters under ordinary and outcome-view contexts, while the realized response and deployment interface remain unchanged.

ALFWorld. ALFWorld (Shridhar et al., 2021) presents a household goal in a text-rendered embodied environment. We cap interaction at 50 turns and report binary success separately on all 140 Seen and 134 Unseen evaluation games. The six task families are Pick and Place, Pick Two and Place, Look at Object, Heat and Place, Cool and Place, and Clean and Place. Training rollouts use the official ALFWorld training split. We do not average the Seen and Unseen splits.

A.3

Training-Only Hindsight Views

The method uses one semantic protocol in both environments. For every decision turn, OutcomeView(τ, k) recovers a turn-aligned local view anchored at the decision’s realized action– observation consequence and serializes it into an augmented scoring context. The realized response tokens remain unchanged, so the ordinary policy and hindsight-conditioned view score the same token sequence. This exact-token correspondence is required by Eq. 4. For turn k, let Pk denote the ordinary prompt before the response. The ordinary scorer receives [Pk ; yk,<t ], whereas the hindsight view receives [Pk+ ; yk,<t ] with Pk+ = Aug(Pk , zk ). Thus, one outcome view zk is shared by all scored tokens of the turn, and the complete recorded response, including its action tokens, is unchanged between views. The outcome view is constructed only after the interaction has produced the corresponding consequence and is discarded after the update. The concrete outcome view is aligned with the

Validation and final evaluation. Training-time validation uses a fixed 128-instance diagnostic subset in each environment. The ALFWorld curve in Figure 3 evaluates this same subset every five optimization steps. Main-table values instead use the complete official evaluation sets above under the shared full-set evaluator. Neither the 128-instance diagnostic runs nor the complete evaluation runs are used to construct training updates. A.2

WebShop

Training Configuration

All systems are trained for 150 optimization steps. Group-relative methods sample eight trajectories for each task instance, and all backbones use their standard instruct templates. We use a learning rate of 1 × 10−6 . Table 3 summarizes the taskdependent settings shared by GRPO and TRIAL. The TRIAL auxiliary coefficient is zero for steps 1–24 and takes the value in Table 3 from step 11

current decision. In WebShop, it contains the page state reached by the current action; in ALFWorld, it contains the text-rendered post-action state. Terminal turns use the available terminal consequence. These are environment-specific serializations of the shared OutcomeView interface rather than different scoring protocols, and all outcome views are discarded after optimization.

Both examples instantiate the same protocol: a turn-aligned view of the local realized consequence is inserted into the scoring context before the unchanged response. The number of native event records needed to serialize that semantic view is determined by the environment’s interaction boundary. A.5

A.4

Hindsight Context Examples

Baseline Implementation

GRPO is implemented in the same Agentic RL stack as TRIAL and uses the same task interface, rollout budget, and final evaluation protocol. SERL is run from its released implementation with immediate environment feedback, its native action-level objective, and the task settings used for our comparison. SDAR, GRPO+OPSD, and RLSD are run from the released SDAR implementation with their method-specific objectives and recommended hyperparameters. We preserve each method’s optimization rule rather than rewriting all baselines as variants of our loss. Every reported method is finally evaluated through the same complete-set evaluator and metric implementation.

The following schematic examples abbreviate the task prompt and history while preserving the semantic placement of the outcome view used during training. In both environments, the text after R ECORDED RESPONSE is copied unchanged into the ordinary and hindsight views. Line breaks inside the schematic responses below are typographic only. ALFWorld. Suppose the task is to place an apple in a target receptacle and the recorded turn takes the apple from a cabinet. The two scoring sequences have the form O RDINARY VIEW: [task; interaction history; current observation] R ECORDED RESPONSE : <think>...</think> <action>take apple 1 from cabinet 1</action>

A.6

Compute and Reproducibility

Training runs use eight H20 GPUs. The training configurations and evaluation protocol above specify the settings needed to interpret the reported results.

H INDSIGHT VIEW: [outcome-view-augmented scoring prompt] P OST- ACTION EVIDENCE : “After this response, the environment returns: You pick up apple 1 from cabinet 1.” R ECORDED RESPONSE : [the identical response above]

Additional Analysis Details

B.1

Learning-Dynamics Construction

Figure 3 contains measured validation points rather than an interpolation of final test results. Panel (a) records success on the fixed 128-game ALFWorld validation set at steps 5, 10, . . . , 150. Panels (b–c) partition the corresponding step-level mechanism logs into non-overlapping five-step windows and report arithmetic means. These training diagnostics are reported separately from the complete-set results in Table 1. At each step, the mean turn score and profile dispersion are computed over the turn rows in that actor update, including both successful and unsuccessful training rollouts. Each five-step window therefore summarizes five consecutive updates rather than five independent training runs.

Here zk is the text-rendered state observed after the recorded action. The hindsight view does not replace the action or generate a revised trajectory; it only rescores the recorded response with its realized consequence visible. WebShop. Suppose the task requests a navy shirt and the current turn opens a candidate product. The corresponding context includes the immediate page reached by that action: O RDINARY VIEW: [task; interaction history; current search page] R ECORDED RESPONSE : <think>...</think> <action>click[product]</action> H INDSIGHT VIEW: [outcome-view-augmented scoring prompt] P OST- ACTION EVIDENCE : “The product page shows color options Navy and Black.” R ECORDED RESPONSE : [the identical response above]

B.2

Allocation-Trace Construction

Figure 4 uses one native training rollout batch. Panel (a) selects a readable 10–25-turn successful 12

Stage

GRPO inference trace

TRIAL inference trace

Goal and dis- Reason: “Check the countertop for a credit card.” Reason: “Check the target, then the nearby table.” covery T1: go to countertop 1 → creditcard 3 is observed. T1: go to dresser 1 → no credit card is present. T2: T2: take creditcard 3 → acquisition succeeds. T3: go to sidetable 1 → creditcards 1 and 2 are observed. go to sidetable 1 → creditcards 1 and 2 are observed. First placement After moving between the sidetable and dresser, the T3: take creditcard 1 → acquisition succeeds. [one policy claims both cards are in inventory. T6: move no-change turn] T5: go to dresser 1 → dresser is creditcard 1 to dresser 1 → “Nothing happens.” · · · reached. T6: move creditcard 1 to dresser 1 → T9: take creditcard 1 → acquisition succeeds. T12: placement succeeds. move creditcard 1 to dresser 1 → placement succeeds. Second-card re- The policy repeatedly reasons that creditcard 2 is at After three failed remote acquisitions, the policy covery the dresser or countertop, despite observing it at the returns to the last observed location. T10: go to sidetable. T13–T49 contain repeated searches and sidetable 1 → creditcard 2 is observed. T11: take failed remote acquisitions; T50 reaches the sidetable creditcard 2 → acquisition succeeds. T13–T14: go only after the turn budget is exhausted. to dresser 1; move creditcard 2 → task completes. Failure at 50 turns; 18 actions produce no state Success at 14 turns; 5 actions produce no state Outcome change. change.

Table 4: Compressed deterministic inference traces for the same ALFWorld task: put two credit cards in the dresser. Quoted reasoning is abbreviated from the generated response; · · · and bracketed text denote omitted turns, not reconstructed actions.

Pick-Two trajectory containing acquisition, ineffective interaction, recovery, and completion. Panel (b) includes all 21 successful Pick-Two trajectories for the five task instances in that batch. Action classes are assigned from the environment action string: take marks acquisition, move or put marks placement, go to marks navigation, and the terminal turn marks completion; remaining actions form inspection/other. Error bars are descriptive SEM across trajectory-level category means and are not used for a significance claim. B.3

Paired Inference Case Study

Table 4 compares deterministic inference traces from GRPO and TRIAL on the same Seen PickTwo game. Both policies receive the same initial state and admissible-action interface. We retain the generated reasoning cue when it explains the next decision, show the realized observation after each displayed action, and mark omitted turns explicitly. The contrast is behavioral: aggregate comparisons remain in Table 1, while Figure 4 analyzes the training-time allocation mechanism.

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning

Analyze & explore related research in Conceptio

Related documents