arXiv CSopen access

CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents

CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents

Yujiang Li*

†

Zhenyu Hou*

†

†

Yi Jing

Jie Tang

Yuxiao Dong

arXiv:2607.05378v1 [cs.LG] 6 Jul 2026

Tsinghua University

Abstract Long-horizon agentic LLMs are increasingly limited by finite context windows, as extended interaction trajectories can exceed the maximum context length before a task is completed. Context compaction offers a natural solution by summarizing previous interaction states and continuing the rollout under a compressed context, but incorporating compaction into reinforcement learning remains underexplored. We propose CompactionRL, a reinforcement learning strategy to train long-horizon agentic LLMs with context compaction. Our approach jointly optimizes task execution and summary generation with token-level loss normalization and crosstrajectory generalized advantage estimation. This design enables the LLM agents to learn from compacted long-horizon trajectories. We train CompactionRL on top of open models and observe consistent performance gains on agentic coding tasks. CompactionRL enables the open GLM-4.5-Air model (106B-A30B) to achieve Pass@1 scores of 66.8% on SWE-bench Verified and 24.5% on Terminal-Bench 2.0, with absolute gains of 7.0 and 3.1 points, respectively. Built upon GLM-4.7Flash (30B-A3B), CompactionRL improves Pass@1 by 5.5 and 6.8 points, reaching 56.0% on SWE-bench Verified and 20.2% on Terminal-Bench 2.0, respectively. CompactionRL is thus deployed in the RL pipeline for training the open GLM-5.2 model (750B-A40B).

1

Introduction

Large language model (LLM) agents are increasingly applied to long-horizon interactive tasks, such as software engineering, terminal-based problem solving, and web interaction [Yao et al., 2023, Nakano et al., 2021, Jimenez et al., 2023]. These tasks require agents to repeatedly reason, act, observe environment feedback, and revise their plans over many steps. As the interaction proceeds, the accumulated history, including tool outputs, intermediate reasoning, error messages, and partial solutions, can exceed the model’s finite context window. Although longer-context models alleviate this issue, scaling context length alone is costly and does not fully solve degraded utilization over long sequences [Beltagy et al., 2020, Dao et al., 2022, Liu et al., 2024, Hsieh et al., 2024]. Thus, long-horizon RL for LLM agents requires mechanisms that allow training to proceed under a fixed context budget. Context compaction provides a natural way to address this limitation. When the history approaches the context limit, earlier interaction history can be summarized into a shorter state, after which the agent resumes from the compacted summary and a small amount of recent context. Related ideas have been explored in language-agent memory, reflection, and long-horizon interaction systems [Shinn et al., 2023, Park et al., 2023, Sun et al., 2025]. However, prior uses of compaction are mostly treated as inference-time heuristics or external memory operations. In RL training, compaction has a more * Equal Contribution. † Work done while YL, ZH, and YJ interned at Z.AI.

Preprint.

Inference w/o compaction Context window

Context budget exhausted

100 Prompt

step 1

step 2

···

step N

GLM-4.7-Flash GLM-4.5-Air

GLM-4.7-Flash + CompactionRL GLM-4.5-Air + CompactionRL

Stopped

Context window Prompt

step 1

step 2

···

step N

Context budget exhausted Compaction

（summarization） Summary

New context window step 1

step 2

···

Accuracy (%)

80

Inference with compaction

+7.0 60

+5.5

40

+6.8

20 End

+3.1

Completed 0

30B

106B

SWE-Verified

Compact past reasoning into a summary, then continue in a new trace.

30B

106B

Terminal-Bench 2.0

Figure 1: Left: Context compaction allows execution to continue under a fixed context window by initiating a new trace from a compressed summary once the context budget is exhausted. Without compaction, execution terminates upon exhaustion of the context budget. Right: CompactionRL consistently outperforms baseline models on SWE-bench Verified and Terminal-Bench 2.0 under compaction evaluation at both GLM-4.7-Flash(30B-A3B) and GLM-4.5-Air(106B-A30B) scales.

fundamental role: once a summary replaces the original history, it determines what information is available for all subsequent actions. Therefore, task success depends not only on the execution policy, but also on the quality of the compaction policy. Existing RL pipelines for LLM agents are not designed for this setting. Recent methods such as GRPO optimize groups of complete rollouts with group-normalized advantages [Shao et al., 2024], while standard long-horizon RL methods typically assume that each trajectory can be processed without changing its context representation. Context compaction breaks these assumptions. A single rollout may be split into a variable number of segments depending on when compaction is triggered, making fixed-size group normalization ill-suited. Moreover, training compacted segments as independent samples can distort both loss weighting and temporal credit assignment, since segments from the same rollout share a task-level reward but differ in length and temporal distance to the final outcome. In this work, we propose CompactionRL, a PPO-based reinforcement learning framework for building long-horizon agentic LLMs with trainable context compaction, as shown in Figure 1. CompactionRL incorporates compaction into rollout collection, and reconstructs the agent context from a summary once context budget is exhausted. Its core strategy is to jointly optimize task execution actions and summarization actions under the same final task reward. This enables RL training horizon to extend beyond the peak context length while keeping the actual working context budget fixed. Our contributions are summarized as follows: • We bring context compaction into long-horizon agentic RL training. Compaction leads to a variable number of rollout segments, therefore, we move from group-wise optimization to a PPO formulation that learns from individual compaction segments. This helps avoid the constraints of group-relative baselines and instead relies on a critic to estimate token-level advantages. • We develop CompactionRL to jointly train trajectory generation and summary generation with a shared task-level reward, making summarization a learned part of the model rather than an inference-time heuristic. To handle compacted rollouts, we use token-level loss normalization to reduce length-induced weighting bias and cross-trajectory GAE to preserve temporal credit assignment across compaction boundaries. • CompactionRL achieves consistent gains on long-horizon agentic coding tasks. Compared with the base model with inference-time compaction, CompactionRL improves GLM-4.7Flash by 5.5 points on SWE-bench Verified and 6.8 points on Terminal-Bench 2.0. It also improves GLM-4.5-Air by 7.0 and 3.1 points on the two coding benchmarks, respectively. These results show that RL-trained compaction can extend the effective training horizon of agentic LLMs without increasing the maximum working context length. 2

2

Related Work

Context Compaction for Long-Horizon Agents. Long-horizon language agents must maintain task-relevant state across extended sequences of reasoning, tool calls, and environment feedback. A standard ReAct-style agent appends the full interaction history to the prompt [Yao et al., 2023], but this makes the context grow with the trajectory and can reduce both efficiency and effective information use in long contexts [Liu et al., 2024]. Prior work has addressed this issue through longer-context modeling and evaluation [Bai et al., 2024, Hsieh et al., 2024], prompt or observation compression [Jiang et al., 2023b,a], and explicit memory or reflection mechanisms for agents [Packer et al., 2023, Shinn et al., 2023, Park et al., 2023, Zhou et al., 2025]. These methods show that compressing or externalizing history can extend the effective horizon of LLM agents. Recent work has begun to incorporate context management into agent training. SUPO [Lu et al., 2025] introduces summarization-based context management into multi-turn RL and optimizes summarization together with tool-use behavior; ReSum [Wu et al., 2025] uses a periodic external summary tool and trains summary-conditioned agents with segmented trajectories; and Context-Folding [Sun et al., 2025] frames context management as a branch-and-fold skill trained with reinforcement learning. However, how to jointly train long-horizon coding agents with both execution and compaction segments remains underexplored. Reinforcement Learning for Language Models. Reinforcement learning has become a central technique for improving language models with outcome feedback[Uesato et al., 2022]. Early RLHF systems commonly adopt PPO with an explicit value function [Schulman et al., 2017, Ouyang et al., 2022], while recent reasoning-oriented LLM training often favors critic-free or group-relative objectives such as RLOO, GRPO, and DAPO [Ahmadian et al., 2024, Shao et al., 2024, DeepSeek-AI, 2025, Hou et al., 2025b, Yu et al., 2025, Zheng et al., 2025] that have shown strong performance on verifiable-reward tasks such as mathematical reasoning and coding[Hou et al., 2025a, Wei et al., 2025, Xu et al., 2025]. At the same time, recent studies show that critic-based methods can still be highly effective when value learning is carefully designed. VC-PPO [Yuan et al., 2025] improves PPO through value pretraining and modified advantage estimation and VAPO [Yue et al., 2025] further demonstrates that a value-based RL framework can achieve strong and stable reasoning performance. These results suggest that critics remain useful for long-horizon credit assignment.

3

Preliminaries

Proximal Policy Optimization (PPO). In reinforcement learning for language models, Proximal Policy Optimization (PPO) [Schulman et al., 2017] stabilizes policy updates with a clipped surrogate objective: J CLIP (θ) = Et [min (ρt (θ)At , clip(ρt (θ), 1 − ϵ, 1 + ϵ)At )] , (1) where ρt (θ) is the importance sampling ratio between the current and rollout policies, At is the advantage estimate, and ϵ is the clipping threshold. The PPO critic additionally learns a value function Vϕ by regressing to return targets:  2  LVF (ϕ) = Et Vϕ (xt ) − R̂t .

(2)

PPO commonly uses Generalized Advantage Estimation (GAE) [Schulman et al., 2015] to compute token-level advantages. For a trajectory with T optimized tokens, the temporal-difference residual is δt = rt + γVϕ (xt+1 ) − Vϕ (xt ),

(3)

with Vϕ (xT +1 ) = 0 at the terminal state. GAE computes AGAE = t

T −t X

(γλ)ℓ δt+ℓ ,

(4)

R̂t = AGAE + Vϕ (xt ). t

(5)

ℓ=0

and the corresponding return target is

3

Cross trajectory GAE

CompactionRL

Token-level loss

Resume context Recent turns

•••

Execution Segment 1

Execution Segment 2

Summary 1

Verify

Recent turns

•••

R

Execution Segment 3

Summary 2

Shared Reward

RL(w/o compaction) Assistant turns

Observation

Verify

•••

Single Execution Segment

Execution Segment

Summary

R

•••

Summary Segment

Figure 2: Overview of CompactionRL. During rollout collection, the agent interacts with the environment under a fixed context budget. When the remaining context budget falls below a compaction threshold Tcomp , the policy generates a summary and resumes from the summary and recent turns. All execution and summary segments are optimized and share the trajectory reward R. During training, we propagate this reward across segment boundaries by correcting the local GAE advantage Aloc s,i with the number of optimized tokens in subsequent segments, N>s , yielding the cross-trajectory advantage bs,i . Standard RL without compaction optimizes each rollout as a single execution segment. A

4

The CompactionRL Method

We present CompactionRL, a reinforcement learning framework for long-horizon language agents under a fixed context budget. The central idea is to make context compaction part of rollout collection: when the interaction history approaches the context limit, the agent produces a compact summary of the previous trajectory and then resumes from a reconstructed context containing the summary and a short history tail of recent interaction. The summary is generated by the trainable policy and is optimized together with ordinary task-execution actions under the final task reward. Figure 2 gives an overview of the framework. 4.1

Trainable Context Compaction

Compaction during Rollout Collection.

We represent the interaction history as

ht = (s, u, z1 , . . . , zt ),

zi = (ai , oi ),

(6)

where s is the system prompt, u is the original user instruction, ai is the assistant response at step i, and oi is the corresponding environment observation. We treat each assistant-observation pair zi as an atomic step, so that tool calls and their feedback are not separated by compaction. Let C denote the context budget and |ht | the token length of the current history. Compaction is triggered when the remaining context budget falls below a threshold Tcomp : C − |ht | < Tcomp .

(7)

Once triggered, we append a fixed summarization instruction qsum to the current history and sample a summary from the policy, St ∼ πθ (· | ht ⊕ qsum ), (8) where qsum asks the model to preserve information necessary for continuing the task, including the original goal, completed actions, important observations, unresolved errors, current state, and plausible next steps. After summary generation, the rollout continues from a reconstructed context h̄t = (s) ⊕ uresume (St ) ⊕ (zt−k+1 , . . . , zt ), 4

(9)

Table 1: Effect of summarization quality under context compaction. The execution agent is fixed to GLM-4.7-Flash, while tested with various summary agents including Qwen3.5-27B[Qwen Team, 2026] and Qwen3-30B-A3B[Yang et al., 2025]. SWE-Verified Acc. denotes pass@1(%) on SWEbench Verified; Summary Count / Trace reports the average number of summaries triggered per trace. Execution Agent

Summary Agent

SWE-Verified Acc.

Summary Count / Trace

GLM-4.7-Flash GLM-4.7-Flash GLM-4.7-Flash

Qwen3.5-27B GLM-4.7-Flash Qwen3-30B-A3B

55.5 50.5 49.0

1.010 1.075 1.126

where uresume (St ) is a fixed template containing the generated summary. The final term keeps the most recent k steps. We use k = 2 by default and reduce k when necessary to ensure that h̄t fits within the context budget. This reconstruction preserves long-range information through St while retaining the most recent environment state exactly. Training Segments. A complete rollout with compaction is naturally partitioned into a sequence of generated-token segments, τ = (σ1 , . . . , σK ), (10) where each segment is either an execution segment or a summarization segment. An execution segment contains tokens produced while solving the task, while a summarization segment contains the summary tokens generated before context reconstruction. Importantly, the summarizer is not an external module: summary tokens are sampled from the same trainable policy and are included in the RL objective. Each rollout receives a final task reward R(τ ) determined by task correctness. We assign this rollout-level reward to all trainable segments from the same rollout. We do not introduce a separate summary-quality reward, since hand-designed summary metrics may not reflect which details are useful for solving the task. Effect of Summarization Quality. Compaction makes task success sensitive to the quality of the summary. A summary that omits a crucial file path, error message, failed command, or partial solution can make subsequent actions ineffective even when the execution policy itself is strong. To quantify this effect, we fix the execution agent and vary only the summary agent. Table 1 shows that changing only the summary agent leads to a large difference in final task performance. The best summarizer improves SWE-Verified accuracy from 49.0 to 55.5, a gain of 6.5 absolute points, while also triggering slightly fewer compactions per trace. This result suggests that compaction is a performance-critical decision process rather than a passive preprocessing step. We therefore train summary generation jointly with task execution. 4.2

Optimization under Compacted Rollouts

Ill-Suited Group-Wise Methods. Context compaction changes the sampling structure of RL data. Group-wise methods such as GRPO sample a fixed group of complete rollouts for each prompt and estimate advantages by normalizing rewards within the group [Shao et al., 2024]. This assumption becomes problematic when a rollout is split by compaction. If each compacted segment is treated PG as an optimization sample, then a group of G rollouts no longer yields G samples, but g=1 Kg segments, where Kg is the number of segments in rollout g. Since all segments from the same rollout share the same final reward, rollouts with more compaction events would be repeated more times in the group statistics and would receive disproportionate weight. Conversely, if group normalization is performed only at the complete-rollout level, it does not provide segment-level advantages for independently optimized execution and summarization segments. For this reason, we instantiate CompactionRL with PPO rather than a group-wise advantage estimator. Its value-function-based advantage estimation avoids reliance on fixed-size reward groups and supports variable numbers of compacted segments, including the case where only one rollout is sampled for a prompt. 5

Token-Level Loss for Length Imbalance. Since the length of agent trajectories and summarization varies significantly, we choose to use token-level loss instead of sequence-level loss to overcome the severe imbalance. Let M denote the set of optimized assistant-token positions in a training batch. For token ys,i in segment σs , conditioned on context xs,i , define the PPO probability ratio ρs,i (θ) =

πθ (ys,i | xs,i ) . πθold (ys,i | xs,i )

We optimize the clipped PPO objective   X 1 bs,i , clip (ρs,i (θ), 1 − ϵ, 1 + ϵ) A bs,i , Lπ = − min ρs,i (θ)A |M|

(11)

(12)

(s,i)∈M

bs,i is the advantage estimate described below. The value model is trained with the standard where A PPO value regression loss. The loss computation is performed over generated tokens rather than over samples. This is important because compaction introduces heterogeneity in both segment count and segment length. Segmentlevel averaging would make a rollout with more compaction events contribute more strongly than a rollout with fewer segments, even when both receive the same task reward. Token-level normalization removes this segment-count bias and gives each trainable token equal weight. Cross-Trajectory Generalized Advantage Estimation. Compacted rollouts also create a temporal credit-assignment issue. Since each segment is optimized independently, a naive segment-level GAE computation places the shared terminal reward at the end of every segment. For earlier segments, this makes the final task outcome appear artificially closer than it is in the original rollout, over-crediting actions or summaries that occur far before task completion. For a segment σs with ns optimized tokens, the local GAE estimator is Aloc s,i =

nX s −i

(γλ)ℓ δs,i+ℓ ,

δs,i = rs,i + γVϕ (xs,i+1 ) − Vϕ (xs,i ),

(13)

ℓ=0

P where i indexes tokens within the segment. Let N>s = j>s nj be the number of optimized tokens generated after segment σs in the same rollout. We apply a trajectory-position correction bs,i = (γλ)N>s Aloc A s,i .

(14)

This correction discounts earlier segments according to the number of subsequent trainable tokens before rollout termination. In particular, if the terminal task reward appears at the last token of each independently optimized segment, then the reward term for token (s, i) receives discount (γλ)N>s +ns −i ,

(15)

which matches its distance to the final outcome in the concatenated compacted rollout.

5

Experiments

5.1

Experimental Setup

Training Details. We conduct experiments with two models at different parameter scales: GLM-4.7Flash and GLM-4.5-Air-SFT. The latter is obtained by supervised fine-tuning GLM-4.5-Air [GLM et al., 2025] on trajectories generated by GLM-4.7. For each model, we initialize the corresponding critic from the same model checkpoint and perform 50 steps of value pretraining on the training dataset before reinforcement learning. We use open-source training data from SWE-Dev [Wang et al., 2025]. For the training of CompactionRL, we use slime [Zhu et al., 2025], an open-source asynchronous RL framework, for reinforcement learning training. For RL training, we employ a global batch size of 128, a group size of 1, a context budget of 64k for GLM-4.7-Flash and 80k for GLM-4.5-Air-SFT. The policy is optimized with Adam using a learning rate of 2 × 10−6 , while the critic is trained with a learning rate of 3 × 10−6 . For advantage estimation, 1 we adopt length-adaptive GAE [Yue et al., 2025] with λ = 1 − αl and α = 1.5, where l denotes 6

Table 2: Main results on SWE-bench Verified and Terminal-Bench 2.0. We report Pass@1 accuracy (%). We compare the base models, standard PPO without compaction, and CompactionRL. Peak Length denotes the maximum working context length available. For each setting, training and evaluation are conducted under the same peak length. Single(×1) evaluation disables compaction and uses one Peak-Length context window as the budget, while compacted (×4) evaluation allows up to three compaction operations, yielding an effective budget of 4× Peak Length. SWE-bench Verified

Terminal-Bench 2.0

Single (×1)

Compacted (×4)

Single (×1)

Compacted (×4)

400k 256k 128k 256k 256k 256k 64k

72.0 66.5 62.0 51.9 45.2 25.2 –

– – – – – – 58.0

31.9 23.9 18.7 14.6 13.5 5.34 –

– – – – – – 27.0

GLM-4.7-Flash (30B-A3B) + RL (w/o compaction) + CompactionRL (ours)

64k 64k 64k

47.5 50.0 43.7

50.5 48.0 56.0

14.6 16.9 16.9

13.4 12.4 20.2

GLM-4.5-Air (106B-A30B) + RL (w/o compaction) + CompactionRL (ours)

80k 80k 80k

57.8 58.3 57.3

59.8 62.5 66.8

17.9 20.2 21.4

21.4 23.6 24.5

Model

Peak Len.

Public reports / baselines GPT-5 mini (2025-08-07) Qwen3-Coder-480B-A35B-Instruct gpt-oss-120b Qwen3-Coder-30B-A3B-Instruct Qwen3-235B-A22B-Instruct-2507 Qwen3-30B-A3B-Instruct-2507 Qwen3.5-35B-A3B Terminus-KIRA scaffold

Note. In all our results, including Qwen3.5-35B-A3B, SWE-bench Verified is evaluated on a random 200-instance subset, while SWE-bench Verified results of public baselines are reported on the full benchmark. The public baselines are taken from model cards or public reports and are included for reference only, as their agent scaffolds may differ from ours.

the response length. To ensure that the critic keeps pace with policy updates, we perform two value model updates and one policy update for each batch, facilitating accurate advantage estimation. Each assistant response is limited to 10,240 tokens and compaction is triggered when the remaining context budget falls below 10,240 tokens, with at most three compaction operations per rollout. Evaluation Setting. We evaluate CompactionRL on SWE-bench Verified [Jimenez et al., 2023] and Terminal-Bench 2.0 [Merrill et al., 2026], using 200 randomly sampled tasks from SWE-bench Verified and the full Terminal-Bench 2.0 set, and report Pass@1 accuracy. All evaluations are conducted in the Harbor [Harbor Framework Team, 2026] environment with the Terminus-KIRA [KRAFTON AI and Ludo Robotics, 2026] agent scaffold. Evaluation hyperparameters include: top-p = 1.0, temperature = 1.0, up to 250 interaction turns and at most three compaction operations per trajectory. For each experiment, we report the mean performance of 2 evaluation runs. 5.2

Main Results

Table 2 compares base models, standard PPO trained without compaction, and CompactionRL under both single-window and compacted evaluation settings on SWE-bench Verified and Terminal-Bench 2.0. Under a fixed peak working context length, CompactionRL achieves the best compacted-inference performance among all variants on both benchmarks. This demonstrates that compaction-aware training improves the model’s ability to act on compacted histories. Standard PPO improves singlewindow execution, but this benefit does not transfer consistently to compacted inference, where it either degrades performance or yields smaller gains than CompactionRL. Meanwhile, CompactionRL does not consistently improve single-window performance, which is expected because disabling compaction creates a mismatch with its training setting and results in a higher overlong rate. Overall, the results show that jointly training task execution and compaction is important for reliably benefiting from context compaction and brings substantial gains under the same context budget. 7

Table 3: Ablation study on context compaction and summary training. We report Pass@1 accuracy (%) on SWE-bench Verified and Terminal-Bench 2.0. Each row corresponds to one training configuration, where Train Budget denotes the effective context budget used during RL training and Summary Train indicates whether summary responses are included in the RL loss. Single(×1) disables compaction, Comp.(×4) enables up to three compaction operations, and Long uses a larger non-compacted context window as a reference. System

Train Summary SWE-bench Verified Terminal-Bench 2.0 Budget Train Single Comp. Single Comp. Long Long (×1) (×4) (×1) (×4)

30B evaluation setting: Single/Comp.(×4) use 64k peak length; Long uses 128k GLM-4.7-Flash(30B-A3B) – + RL (w/o compaction)-64k 64k + RL (w/o compaction)-128k 128k + CompactionRL(w/o sum.) 64k×4 + CompactionRL(ours) 64k×4

– ✗ ✗ ✗ ✓

47.5 50.0 48.3 52.5 43.7

50.5 48.0 52.5 54.5 56.0

53.5 48.5 59.0 50.2 49.0

14.6 16.9 11.8 9.0 16.9

13.4 12.4 23.6 12.4 20.2

16.9 12.4 14.6 11.2 16.9

21.4 23.6 23.0 21.5 24.5

20.8 21.1 23.6 20.2 22.5

106B evaluation setting: Single/Comp.(×4) use 80k peak length; Long uses 160k GLM-4.5-Air(106B-A30B) – + RL (w/o compaction)-80k 80k + RL (w/o compaction)-160k 160k + CompactionRL(w/o sum.) 80k×4 + CompactionRL(ours) 80k×4

5.3

– ✗ ✗ ✗ ✓

57.8 58.3 63.0 54.5 57.3

59.8 62.5 64.5 64.5 66.8

59.5 61.8 64.0 61.6 62.1

17.9 20.2 20.8 17.6 21.4

Ablation Studies

Effect of Summary Training. We further ablate the role of compaction during reinforcement learning and the effect of optimizing the summarizer. Specifically, we compare CompactionRL with no-compaction RL under matched and extended context budgets, and consider a variant that enables compaction during training but masks out the loss on summary-response turns. This separates the effect of exposing the policy to compacted histories from the benefit of updating the summarizer. Table 3 shows that training without compaction under a longer context window provides a strong reference, as it simulates access to a sufficiently long context. While keeping the peak context length short, compaction-aware training remains competitive with this longer-context baseline and often surpasses it, with a similar trend across both model scales. The comparison with the nosummary-training variant further shows that including summary responses in the RL objective consistently improves compacted inference performance across both model scales and benchmarks, demonstrating the benefit of directly optimizing summarization during compaction-aware training. These results indicate that CompactionRL effectively extends the usable context horizon through trainable compaction. Effect of Token-Level Loss and Cross-Trajectory GAE. We further ablate the two optimization components designed for compacted trajectories using GLM-4.5-Air-SFT: token-level loss normalization and cross-trajectory GAE. As shown in Table 4, removing either component degrades performance, confirming that both loss weighting and cross-segment credit assignment are important for compaction-aware RL. A larger degradation is observed when token-level loss normalization is removed, suggesting that correcting the optimization bias induced by variable segment counts and lengths is particularly critical. 5.4

Analysis

Compaction Behavior. Figure 3 analyzes compaction behavior in the GLM-4.5-Air setting under 80k×4 evaluation. Figure 3(a,b) shows that CompactionRL requires fewer compactions and tool calls than GLM-4.5-Air-SFT and the no-summary-training variant but more than standard RL trained without compaction. This indicates that CompactionRL learns to exploit the extended context horizon enabled by compaction as well as a higher interaction efficiency, suggesting that the improvement is 8

Table 4: Ablation study on token-level loss normalization and cross-trajectory GAE using GLM-4.5Air-SFT. All variants are evaluated under the compaction enabled 80k×4 setting. We report Pass@1 accuracy on SWE-bench Verified and Terminal-Bench 2.0. System

SWE-bench Verified

Terminal-Bench 2.0

59.8 66.8 60.0 63.0

21.4 24.5 21.3 22.5

0.6

Average Compaction Times 0.58 0.47 0.36

0.4 0.2 0.0

4.5

GLM

0.21

0.26

-Air L-80k -160k Sum.) mpRL R RL (w/o Co pRL m o C

(a) Average compactions

Average Tool Calls

125 100

90.8

83.4

75 50

48.2

63.5

60.7

25 0

Compacted-Task Accuracy

100

-Air L-80k -160k Sum.) mpRL R RL (w/o Co pRL m o C

-4.5 GLM

(b) Average tool call counts

Pass@1(%)

0.8

Tool Call / Trace

Compaction Count / Trace

GLM-4.5-Air + CompactionRL − w/o token-level loss − w/o cross-trajectory GAE

80 60

35.4

40

45.9 42.4 47.7 29.0

20 0

-Air L-80k -160k Sum.) mpRL R RL (w/o Co pRL m o C

-4.5 GLM

(c) Compacted-task accuracy

Figure 3: Behavior comparison on SWE-bench Verified in the GLM-4.5-Air setting under 80k×4 compacted evaluation. We compare the base model(GLM-4.5-Air-SFT), standard RL without compaction trained at 80k and 160k (RL-80k and RL-160k respectively), CompactionRL without summary training, and CompactionRL. Here, we denote a trace as all compaction segments generated to solve a single task. (a) Average number of compaction operations per trace. (b) Average number of tool calls per trace. (c) Pass@1 accuracy on compaction triggering tasks in each evaluation run.

not attributable to longer interaction traces alone; rather, trained summaries help retain task-relevant information and reduce redundant re-exploration after compaction. Figure 3(c) reports pass@1 on tasks that trigger compaction. CompactionRL achieves the highest accuracy on this subset, indicating that jointly training execution and summarization improve reliable continuation from compacted histories. Training Dynamics. Figure 4(a) analyzes the change of summary length during GLM-4.5-Air-SFT CompactionRL training. Compared with the no-summary-training variant with decreasing summary length, CompactionRL produces increasingly longer and more detailed summaries when summary responses are included in the RL objective. This results in more complete and actionable compacted states that better preserve implementation-relevant context and continuation state. This suggests that optimizing summary responses encourages the model to retain useful detailed information for post-compaction continuation, rather than merely recording high-level task progress. Figure 4(b) further shows that the reasoning tokens of CompactionRL increase throughout training, whereas the no-summary-training and no-compaction variants decrease. This indicates that learned compaction effectively expands the usable context window, allowing for more reasoning budget. Figure 4(c) shows that CompactionRL also exhibits a slower entropy increase, suggesting more controlled policy optimization.

6

Conclusion

We presented CompactionRL, a PPO-based framework that trains long-horizon agents with context compaction by optimizing execution and summary tokens under the same task reward. With tokenlevel loss normalization and cross-trajectory GAE, CompactionRL improves SWE-bench Verified and Terminal-Bench 2.0 performance under a fixed peak context budget, while further ablations show that summary training produces more detailed compacted states and is important for these gains. Overall, our results demonstrate the effectiveness of incorporating context compaction into RL training for long-horizon agents, rather than treating it solely as an inference-time heuristic. 9

2200 CompactionRL CompactionRL(w/o sum.)

2000 1800 1600

0.60

CompactionRL CompactionRL(w/o sum.) RL(w/o compaction)

210 180

0.52

150 120 90

20

40

60

Training Steps

(a) Summary length

80

0.48 0.44 0.40 0.36 0.32

60 0

CompactionRL CompactionRL(w/o sum.) RL(w/o compaction)

0.56

Entropy

Reasoning Tokens

Summary Length

2400

0

20

40

60

Training Steps

(b) Reasoning tokens per turn

80

0

20

40

60

Training Steps

80

(c) Entropy

Figure 4: Training dynamics of GLM-4.5-Air CompactionRL. (a) Summary optimization leads to increasingly longer and more detailed summaries. (b) CompactionRL increases reasoning tokens per turn. (c) Policy entropy. Limitations. Despite its effectiveness, CompactionRL is trained for compaction-enabled execution, and its gains do not consistently transfer to single-window evaluation when compaction is disabled. This indicates a train–test mismatch and limits its direct use in settings without test-time compaction. In addition, cross-trajectory GAE remains an approximation to full-trajectory credit assignment and may not fully capture the long-term effects of early summaries across multiple compaction boundaries. Finally, our experiments mainly focus on code-oriented long-horizon benchmarks; extending this approach to broader agent domains with different observation structures and reward signals remains an important direction for future work.

References Arash Ahmadian et al. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2024. Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, volume 35, pages 16344–16359, 2022. DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Team GLM, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bowen Xu, Can Huang, Casey Zhao, Changpeng Cai, Chao Yu, Chen Li, Chendi Ge, Chenghua Huang, Chenhui Zhang, Chenxi Xu, Chenzheng Zhu, Chuang Li, Congfeng Yin, Daoyan Lin, Dayong Yang, Dazhi Jiang, Ding Ai, Erle Zhu, Fei Wang, Gengzheng Pan, Guo Wang, Hailong Sun, Haitao Li, Haiyang Li, Haiyi Hu, Hanyu Zhang, Hao Peng, Hao Tai, Haoke Zhang, Haoran Wang, Haoyu Yang, He Liu, He Zhao, Hongwei Liu, Hongxi Yan, Huan Liu, Huilong Chen, Ji Li, Jiajing Zhao, Jiamin Ren, Jian Jiao, Jiani Zhao, Jianyang Yan, Jiaqi Wang, Jiayi Gui, Jiayue Zhao, Jie Liu, Jijie Li, Jing Li, Jing Lu, Jingsen Wang, Jingwei Yuan, Jingxuan Li, Jingzhao Du, Jinhua Du, Jinxin Liu, Junkai Zhi, Junli Gao, Ke Wang, Lekang Yang, Liang Xu, Lin Fan, Lindong Wu, Lintao Ding, Lu Wang, Man Zhang, Minghao Li, Minghuan Xu, Mingming Zhao, Mingshu Zhai, Pengfan Du, Qian Dong, Shangde Lei, Shangqing Tu, Shangtong Yang, Shaoyou Lu, Shijie Li, Shuang Li, Shuang-Li, Shuxun Yang, Sibo Yi, Tianshu Yu, Wei Tian, Weihan Wang, Wenbo Yu, Weng Lam Tam, Wenjie Liang, Wentao Liu, Xiao Wang, Xiaohan Jia, Xiaotao Gu, Xiaoying Ling, Xin Wang, Xing Fan, Xingru Pan, Xinyuan Zhang, Xinze Zhang, Xiuqing Fu, Xunkai Zhang, Yabo 10

Xu, Yandong Wu, Yida Lu, Yidong Wang, Yilin Zhou, Yiming Pan, Ying Zhang, Yingli Wang, Yingru Li, Yinpei Su, Yipeng Geng, Yitong Zhu, Yongkun Yang, Yuhang Li, Yuhao Wu, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yuxuan Zhang, Zezhen Liu, Zhen Yang, Zhengda Zhou, Zhongpei Qiao, Zhuoer Feng, Zhuorui Liu, Zichen Zhang, Zihan Wang, Zijun Yao, Zikang Wang, Ziqiang Liu, Ziwei Chai, Zixuan Li, Zuodong Zhao, Wenguang Chen, Jidong Zhai, Bin Xu, Minlie Huang, Hongning Wang, Juanzi Li, Yuxiao Dong, and Jie Tang. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025. URL https://arxiv.org/abs/2508.06471. Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments, 2026. URL https://doi.org/10.5281/zenodo.20953922. Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong. Treerl: Llm reinforcement learning with on-policy tree search, 2025a. URL https://arxiv.org/abs/2506.11902. Zhenyu Hou, Xin Lv, Rui Lu, Jiajie Zhang, Yujiang Li, Zijun Yao, Juanzi Li, Jie Tang, and Yuxiao Dong. T1: Advancing language model reasoning through reinforcement learning and inference scaling, 2025b. URL https://arxiv.org/abs/2501.11651. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839, 2023a. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736, 2023b. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. KRAFTON AI and Ludo Robotics. Terminus-kira: Boosting frontier model performance on terminalbench with minimal harness, 2026. URL https://github.com/krafton-ai/kira. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. Miao Lu, Weiwei Sun, Weihua Du, Zhan Ling, Xuesong Yao, Kang Liu, and Jiecao Chen. Scaling LLM multi-turn RL with end-to-end summarization-based context management. arXiv preprint arXiv:2510.06727, 2025. doi: 10.48550/arXiv.2510.06727. URL https://arxiv.org/abs/ 2510.06727. Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026. URL https://arxiv.org/ abs/2601.11868. 11

Reiichiro Nakano et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. Long Ouyang et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744, 2022. Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems. arXiv preprint arXiv:2310.08560, 2023. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen. ai/blog?id=qwen3.5. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. doi: 10.48550/arXiv.1707.06347. URL https://arxiv.org/abs/1707.06347. Zhihong Shao et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2023. Weiwei Sun, Miao Lu, Zhan Ling, Kang Liu, Xuesong Yao, Yiming Yang, and Jiecao Chen. Scaling long-horizon llm agent via context-folding. arXiv preprint arXiv:2510.11967, 2025. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022. URL https://arxiv.org/abs/2211.14275. Haoran Wang, Zhenyu Hou, Yao Wei, Jie Tang, and Yuxiao Dong. Swe-dev: Building software engineering agents with training and inference scaling. In Findings of the Association for Computational Linguistics: ACL 2025, pages 3742–3761, 2025. Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution, 2025. URL https://arxiv.org/abs/2502. 18449. Xixi Wu, Kuan Li, Yida Zhao, Liwen Zhang, Litu Ou, Huifeng Yin, Zhongwang Zhang, Xinmiao Yu, Dingchu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Minhao Cheng, Shuai Wang, Hong Cheng, and Jingren Zhou. ReSum: Unlocking long-horizon search intelligence via context summarization. arXiv preprint arXiv:2509.13313, 2025. doi: 10.48550/arXiv.2509.13313. URL https://arxiv.org/abs/2509.13313. Yifan Xu, Xiao Liu, Xinghan Liu, Jiaqi Fu, Hanchen Zhang, Bohao Jing, Shudan Zhang, Yuting Wang, Wenyi Zhao, and Yuxiao Dong. Mobilerl: Online agentic reinforcement learning for mobile gui agents, 2025. URL https://arxiv.org/abs/2509.18119. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X. 12

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What’s behind PPO’s collapse in long-CoT? value optimization holds the secret. arXiv preprint arXiv:2503.01491, 2025. doi: 10.48550/arXiv.2503.01491. URL https://arxiv.org/abs/2503.01491. Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025. Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. Mem1: Learning to synergize memory and reasoning for efficient long-horizon agents, 2025. URL https://arxiv.org/abs/2506.15841. Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. slime: An llm post-training framework for rl scaling. https://github.com/THUDM/slime, 2025. GitHub repository. Corresponding author: Xin Lv.

13