Risk Averse Value Expansion for Sample Efficient and Robust Policy Learning

Bo Zhou, Fan Wang, Hongsheng Zeng, Hao Tian

Keywords: reinforcement learning, model-based RL, risk-sensitive, sample efficiency

TL;DR: We extend the model-based value expansion methods with risk-averse learning and achieve state-of-the-art results on challenging continuous control benchmarks.

Abstract: Model-based Reinforcement Learning(RL) has shown great advantage in sample-efficiency, but suffers from poor asymptotic performance and high inference cost. A promising direction is to combine model-based reinforcement learning with model-free reinforcement learning, such as model-based value expansion(MVE). However, the previous methods do not take into account the stochastic character of the environment, thus still suffers from higher function approximation errors. As a result, they tend to fall behind the best model-free algorithms in some challenging scenarios. We propose a novel Hybrid-RL method, which is developed from MVE, namely the Risk Averse Value Expansion(RAVE). In the proposed method, we use an ensemble of probabilistic models for environment modeling to generate imaginative rollouts, based on which we further introduce the aversion of risks by seeking the lower confidence bound of the estimation. Experiments on different environments including MuJoCo and robo-school show that RAVE yields state-of-the-art performance. Also we found that it greatly prevented some catastrophic consequences such as falling down and thus reduced the variance of the rewards.