Risk-Averse Offline Reinforcement Learning

Núria Armengol Urpí, Sebastian Curi, Andreas Krause

Keywords: offline, reinforcement learning, risk-averse, risk sensitive, robust, safety, safe

Abstract: Training Reinforcement Learning (RL) agents in high-stakes applications might be too prohibitive due to the risk associated to exploration. Thus, the agent can only use data previously collected by safe policies. While previous work considers optimizing the average performance using offline data, we focus on optimizing a risk-averse criteria, namely the CVaR. In particular, we present the Offline Risk-Averse Actor-Critic (O-RAAC), a model-free RL algorithm that is able to learn risk-averse policies in a fully offline setting. We show that O-RAAC learns policies with higher CVaR than risk-neutral approaches in different robot control tasks. Furthermore, considering risk-averse criteria guarantees distributional robustness of the average performance with respect to particular distribution shifts. We demonstrate empirically that in the presence of natural distribution-shifts, O-RAAC learns policies with good average performance.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We propose the first risk-averse reinforcement learning algorithm in the fully offline setting.

Code: [ nuria95/O-RAAC](https://github.com/nuria95/O-RAAC)

Data: [MuJoCo](https://paperswithcode.com/dataset/mujoco)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2102.05371/code)