Official Review · ICLR 2025

The paper presents several contributions in the context of offline preference-based reinforcement learning: - a query selection technique to reduce the number of queries asked to a human - a dynamic discounting approach to account for higher uncertainty of returns - a theoretical analysis to prove the efficiency of the proposed approach The paper presents two new techniques for preference-based reinforcement learning: one for selecting queries and one for handling the uncertainty of value estimation. I believe that both could also be useful in the online reinforcement learning setting. The authors may want to comment on that point in the paper. The paper also provides a theoretical analysis assuming that queries can be generated online, although policy training is still performed in the offline reinforcement learning setting. I believe that the related work should mention reinforcement learning from human feedback (RLHF) since the term "RLHF" has recently become more popular than the term "preference-based reinforcement learning". The formalization is a bit strange, may be overly complexed, and may contain some errors, e.g.: In the presentation of MDPs, why is it important to use the non-stationary definition? Why do we need both Bellman operators? Why are the returns in (6) not discounted? What's the impact of learning a return model vs a learning a reward model? Also, do you learn a return function instead of a reward function in the experiments? If return models are used, why not use "return network" in the whole paper and notably in Algorithm 1 to make things less confusing? On the comment about the query selection criterion, the authors state that their proposition is scale sensitive. Isn't it also the case for the variance-based criterion? The theoretical analysis considers a different algorithm compared to Algorithm 1. In addition, the setting for the theoretical analysis is a bit strange to me. The authors assume that online interaction is possible for querying, but not for policy learning. The authors should discuss this point in more details. Does the theoretical analysis really justifies their proposed criterion in Algorithm 1? Minor issues: Citations between parentheses should not be part of a sentence. With out -> Without The discount factor is missing in (2) Line 146: the optimal policy -> an optimal policy (since it may not be unique) Definition 2: please recall the definition of \alpha'-independent Line 202: Shall R be \theta? Line 252: a space is missing before Intuitively Line 265: r_\theta_1 -> r_\theta_i Line 272: imitaition -> imitation Line 329: \hat q_1 -> \hat q_R Here are my main questions: 1) What's the impact of learning a return model vs a learning a reward model? Also, do you learn a return function instead of a reward function in the experiments? If return models are used, why not use "return network" in the whole paper and notably in Algorithm 1 to make things less confusing? 2) On the comment about the que