Efficient and scalable reinforcement learning via hypermodel

Yingru Li, Jiawei Xu, Lei Han, Zhi-Quan Luo

Keywords: Reinforcement Learning; exploration; sample efficiency; computation efficiency

Abstract:

Data-efficient reinforcement learning(RL) requires deep exploration.
Thompson sampling is a principled method for deep exploration in reinforcement learning.
However, Thompson sampling need to track the degree of uncertainty by maintaining the posterior distribution of models, which is computationally feasible only in simple environments with restrictive assumptions.
A key problem in modern RL is how to develop data and computation efficient algorithm that is scalable to large-scale complex environments.
We develop a principled framework, called HyperFQI, to tackle both the computation and data efficiency issues.
HyperFQI can be regarded as approximate Thompson sampling for reinforcement learning based on hypermodel. Hypermodel in this context serves as the role for uncertainty estimation of action-value function.
HyperFQI demonstrates its ability for efficient and scalable deep exploration in DeepSea benchmark with large state space.
HyperFQI also achieves super-human performance in Atari benchmark with 2M interactions with low computation costs.
We also give a rigorous performance analysis for the proposed method, justifying its computation and data efficiency.
To the best of knowledge, this is the first principled RL algorithm that is provably efficient and also practically scalable to complex environments such as Arcade learning environment that requires deep networks for pixel-based control.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7463