Official Review · ICLR 2026

This paper presents in-context preference reinforcement learning (ICPRL), a form of multi-task reinforcement learning where the adaptation to different tasks is done in-context (ie. changing the inputs to the transformer model), and the reward is derived (implicitly or explicitly) from binary preferences. The paper proposes two implementations of ICPRL: in the first one the preferences are considered over state-action pairs (I-PRL), in the second one the preferences are considered over trajectories (T-PRL). Additionally, three pertaining strategies for I-PRL and T-PRL are explored: 1) DP2T where the preferences are simply provided in context, 2) ICPO (only for I-PRL) where preferences are used to directly optimise the transformer model (analogously to Direct Preference Optimisation), and 3) ICRG (only for T-PRL) where a multi-task reward function is learnt from trajectory preferences. Experiments with MetaWorld and DarkRoom show that I-PRL+ICPO and T-PRL+ICRG are competitive with the ICRL baseline, but without the need for explicit reward functions or high-quality reward transitions. * The paper is well written, provides a good motivation, and is easy to follow (though see nitpicks below). * The derivation of ICPO is very interesting and so is its connection to DPO. * The paper tackles an challenging problem (multi-task RL) through creative means (mixing in-context learning with preference-based reinforcement learning) * The theoretical derivation of the paper is _very_ thorough with many useful appendices (though note I did not have time to review all of the appendices). * The paper explores thoroughly two very different approaches to in-context preference reinforcement learning: I-PRL and T-PRL. * **W1**: The form of multi-task learning employed in the experiments is too narrow. For both DarkRoom and MetaWorld, "multi-task" boils down to different initial and goal states. Indeed, it was precisely this issue that motivated MetaWorld [1] (quoting from the abstract, emphasis mine): > However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used meta-reinforcement learning benchmark uses different running velocities for a simulated robot as different tasks. **When policies are meta-trained on such narrow task distributions, they cannot possibly generalize to more quickly acquire entirely new tasks**. Therefore, if the aim of these methods is enable faster acquisition of entirely new behaviors, we **must evaluate them on task distributions that are sufficiently broad** to enable generalization to new behaviors. The current experiments on MetaWorld are a necessary baseline, but the impact of ICPRL would be much more significant if the method worked well on the MT-10 portion of MetaWorld tasks. * **W2**: Though the related works section is really well redacted, it is missing a discussion of how ICPRL differs from goal-conditioned reinforcement learning. Sim