Official Review · NEURIPS 2024

This paper introduces a novel framework called Empirical MDP Iteration (EMIT) to improve the stability and performance of reinforcement learning algorithms. Traditional reinforcement learning algorithms optimize a Markov Decision Process (MDP) using the Bellman equation, which can lead to unstable optimization when function approximation is used. And the EMIT framework addresses this by constructing a sequence of empirical MDPs from the replay memory and using an in-sample Bellman update to learn an estimated Q-function, denoted as $\widehat{Q}$. As claimed in the paper, this method restricts updates to in-sample data, ensuring convergence to a unique optimal $\widehat{Q}$ function and inducing monotonic policy improvement. The paper demonstrates that EMIT can be integrated with existing reinforcement learning algorithms like DQN and TD3, acting as a regularizer to enhance their performance. The experimental results on Atari and MuJoCo benchmarks show that EMIT significantly reduces estimation errors and improves the performance of these algorithms. The authors provide theoretical analysis and extensive experiments to support their claims, highlighting the advantages of in-sample bootstrapping over traditional Bellman updates in reducing estimation errors and improving policy learning stability. 1. **Comprehensive Experimental Evaluation:** The paper presents extensive experimental results on both discrete action space environments (Atari) and continuous control tasks (MuJoCo). The diverse set of benchmarks provides a solid empirical foundation to validate the effectiveness of EMIT across different types of reinforcement learning tasks. 2. **Improved Stability and Performance:** The empirical results show that EMIT significantly enhances the stability and performance of existing reinforcement learning algorithms like DQN and TD3. By integrating EMIT as a regularizer, the paper demonstrates a clear reduction in estimation errors and notable policy improvements across various benchmarks. **Limited Novelty in Core Ideas:** While the EMIT framework introduces in-sample bootstrapping for empirical MDPs, similar ideas have been explored in offline reinforcement learning and distributional reinforcement learning. The paper would benefit from a deeper differentiation from existing works, such as Implicit Q-Learning and In-Sample Actor-Critic. The authors should elaborate on how EMIT offers significant advancements over these methods beyond just integrating with online reinforcement learning algorithms. I recommend the authors clearly delineate the unique contributions of EMIT compared to existing methods like Implicit Q-Learning and In-Sample Actor-Critic. Providing a comprehensive discussion on the theoretical and practical advancements offered by EMIT would strengthen the paper’s claim of novelty. **Scalability Concerns:** The proposed framework requires maintaining and updating two Q-functions (Q and $\widehat{Q}$) simultaneously, potentially do