Official Review · ICLR 2026

This paper proposes Decomposed Policy Optimization (DePO), which decomposes long-context policy optimization for reinforcement learning (RL) into two parts: learning a policy on short-context environments, and then using knowledge distillation to transfer it to long-context environments. The RL model can thus learn robust local reasoning on the short-context datasets. The paper further introduces a variant of DePO, called DePO-IW, that formulates the distillation process as an off-policy reinforcement learning one. The paper presents extensive numerical comparisons to various types of baseline algorithms for long contexts, on ten different datasets, showing that integrating DePO or DePO-IW into existing RL methods generally increases their accuracy and yields higher accuracy than supervised fine-tuning or preference optimization methods. + The experimental results seem quite thorough, covering ten datasets and multiple supervised fine-tuning, preference optimization, and reinforcement learning baselines. In most of these comparisons, DePO and its variant DePO-IW improve the performance of these baselines. + The paper addresses an important problem of handling long contexts, and it can be integrated into existing reinforcement learning methods for long-context queries. --The intuition behind why we would expect any reasoning learned on short contexts to be useful for longer contexts is explained only very briefly, with the idea that local reasoning can be learned on short contexts without being confounded by long contexts. Given that this is the entire premise of DePO, I’d expect a more thorough discussion motivating the method and explaining why it would be expected to work. --The technical challenges of DePO are not explained well. Beyond the insight that we can transfer reasoning learned on short contexts to long contexts, the paper does not seem to have any technical novelty. The reinforcement learning formulations in Equations (3) to (7) are taken from past work, and that in Equation (7) is almost the same as the formulation in Equation (3), yielding little additional insight. -- Many of the details of knowledge distillation (KD) are missing from the paper. For example, how often is KD performed? Only once, after the reinforcement learning model is trained on short contexts? Can any KD algorithm be used, or does the paper need to adapt existing methods to this setting? 1) How long does the KD algorithm take to run in practice? Does this represent significant computational overhead compared to training the reinforcement learning methods? 2) In Table 3, neither DePO nor DePO-IW consistently outperform each other. Under which circumstances would a user want to use DePO-IW instead of DePO? Are there particular problem characteristics that might indicate whether one or the other would perform better? 3) Lemma 3.1 does not seem to use any specifics of the long- or short-context settings, and appears to apply to any scenario in which we appl