Official Review · NEURIPS 2023

This paper introduces the ANQ (Approximate Nearest Neighbor Q-Learning) framework, which aims to provide explainability in reinforcement learning models. ANQ combines neural networks for high performance and memory-based structures for explainability, offering a promising solution for domains like autonomous driving, quantitative trading, and healthcare. The paper discusses the challenges of explainability in reinforcement learning and how ANQ addresses them. It also presents the Sim-Encoder contrastive learning used in ANQ's state representation and provides insights into the evaluations of ANQ on MuJoCo continuous control tasks and its effectiveness in solving continuous tasks. Overall, the paper presents a novel approach to reinforcement learning that balances performance and transparency, making it suitable for real-world applications. Some strengths of this paper include: 1. Novelty: The ANQ framework is a novel approach to reinforcement learning that combines neural networks and memory-based structures to provide explainability in decision-making processes. This approach is different from traditional reinforcement learning methods that focus solely on performance. 2. Explainability: The paper addresses the challenge of explainability in reinforcement learning models, which is crucial for real-world applications where transparency and trustworthiness are essential. ANQ's "data is policy" design principle ensures that the model's decisions are explainable and interpretable. 3. Sim-Encoder contrastive learning: The paper introduces the Sim-Encoder contrastive learning approach for state representation, which demonstrates its effectiveness in memory retrieval learning tasks. This approach enhances ANQ's performance and explainability. 4. Evaluations: The paper provides insights into the evaluations of ANQ on MuJoCo continuous control tasks and its effectiveness in solving continuous tasks. The results show that ANQ outperforms traditional reinforcement learning methods while maintaining explainability. 5. Real-world applications: The paper highlights the potential of ANQ in domains like autonomous driving, quantitative trading, and healthcare, making it suitable for real-world applications. Some potential weaknesses of this paper include: 1. Despite the advantage of interpretability, the performance of this framework is still far from that of SOTA RL algorithm. I think it would be better if this framework could have similar performance to SOTA RL method. 2. Lack of real-world case studies: Although the paper highlights the potential of ANQ in various domains, it does not provide specific real-world case studies or examples to demonstrate the practical application and effectiveness of the framework. Including such case studies would strengthen the paper's claims and provide more concrete evidence of ANQ's utility in real-world scenarios. 3. No comparison was made with other interpretable reinforcement learning algorithms.