Inverse Constrained Reinforcement Learning

Shehryar Malik, Usman Anwar, Alireza Aghasi, Ali Ahmed

Keywords: Constrained reinforcement learning, constrain inference, safe reinforcement learning

Abstract: Standard reinforcement learning (RL) algorithms train agents to maximize given reward functions. However, many real-world applications of RL require agents to also satisfy certain constraints which may, for example, be motivated by safety concerns. Constrained RL algorithms approach this problem by training agents to maximize given reward functions while respecting \textit{explicitly} defined constraints. However, in many cases, manually designing accurate constraints is a challenging task. In this work, given a reward function and a set of demonstrations from an expert that maximizes this reward function while respecting \textit{unknown} constraints, we propose a framework to learn the most likely constraints that the expert respects. We then train agents to maximize the given reward function subject to the learned constraints. Previous works in this regard have either mainly been restricted to tabular settings or specific types of constraints or assume knowledge of transition dynamics of the environment. In contrast, we empirically show that our framework is able to learn arbitrary \textit{Markovian} constraints in high-dimensions in a model-free setting.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: Using reward function and expert demonstrations, learn the constraints present in the environment.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2011.09999/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=3kFQksN8Xd