ANQ: Approximate Nearest-Neighbor Q Learning

Yiwen Chen, Chuheng Zhang, ZEDONG ZHANG, Li Zhao, Jiang Bian, HAOFENG LIU, Jiayi Tan, Marcelo H Ang Jr

Keywords: reinforcement learning, explainable reinforcement learning, semi-parametric learning, Memory-based learning

TL;DR: We introduce ANQ, a semi-parametric reinforcement learning framework for explainable decision-making in domains like autonomous driving and healthcare, leveraging neural networks for performance and memory-based structures for explainability.

Abstract: In specific domains such as autonomous driving, quantitative trading, and healthcare, explainability is crucial for developing ethical, responsible, and trustworthy reinforcement learning (RL) models. Although many deep RL algorithms have attained remarkable performance, the resulting policies are often neural networks that lack explainability, rendering them unsuitable for real-world deployment. To tackle this challenge, we introduce a novel semi-parametric reinforcement learning framework, dubbed ANQ (Approximate Nearest Neighbor Q-Learning), which capitalizes on neural networks as encoders for high performance and memory-based structures for explainability. Furthermore, we propose the Sim-Encoder contrastive learning as a component of ANQ for state representation. Our evaluations on MuJoCo continuous control tasks validate the efficacy of ANQ in solving continuous tasks while offering an explainable decision-making process.

Submission Number: 13536