Official Review · ICLR 2025

This paper proposes a policy gradient algorithm for general utility reinforcement learning where an estimation of occupancy measure is involved to address the challenge caused by large state-action spaces. Provided theoretical analysis covers both statistical and optimization aspects. Additionally, numerical results are presented to demonstrate the algorithm's effectiveness in tasks involving large state-action spaces. This paper identifies a challenge in General Utility Reinforcement Learning/Convex Reinforcement Learning caused by large state-action spaces and proposes a method to address it. The motivation is well-founded, and the proposed method is sound and natural, i.e. utilizing a maximum likelihood estimator to estimate the occupancy measure. The theoretical analysis is comprehensive, and the paper includes a reasonably thorough comparison with closely related existing works in the Appendix. The structure is logical, with ideas presented in a clear and coherent manner. 1. The primary weakness seems to be the limited novelty of the idea. Although the authors provide a comparison with a related work [1] in the Appendix, the advantages of the proposed method over [1] are not immediately clear. For example, it would be beneficial to include a more detailed discussion of the assumptions required in this work versus those in [1], clarifying which are weaker, stronger, or more practical for specific problem settings. Additionally, highlighting the advantages of the proposed model-free algorithm over the model-based approach in [1] would also be helpful. I think these would significantly improve the paper by more clearly demonstrating its unique contributions. 2. Based on the content in the Appendix and my observations, the technical theoretical analysis appears to draw upon existing results. The statistical analysis is derived from [2], while the optimization analysis is based on [3], with minor adaptations to handle estimation error. However, this is not a critical issue if the authors address the concerns outlined in the Weakness 1, as this would help establish the paper’s distinct contributions more effectively. [1] Mutti, M., De Santi, R., De Bartolomeis, P., & Restelli, M. (2023). Convex reinforcement learning in finite trials. Journal of Machine Learning Research, 24(250), 1-42. [2] Huang, A., Chen, J., & Jiang, N. (2023, July). Reinforcement learning in low-rank mdps with density features. In International Conference on Machine Learning (pp. 13710-13752). PMLR. [3] Barakat, A., Fatkhullin, I., & He, N. (2023, July). Reinforcement learning with general utilities: Simpler variance reduction and large state-action space. In International Conference on Machine Learning (pp. 1753-1800). PMLR. 1. It would be helpful if the authors could clarify the additional assumptions or new technical tools required for proving global convergence compared to [1]. 2. Compared to [2], I am wondering if there exists any sacrifice in the results when yo