Block Contextual MDPs for Continual Learning

Shagun Sodhani, Franziska Meier, Joelle Pineau, Amy Zhang

Keywords: Reinforcement Learning, MDP, Block Contextual MDP, Continual Learning

Abstract: In reinforcement learning (RL), when defining a Markov Decision Process (MDP), the environment dynamics is implicitly assumed to be stationary. This assumption of stationarity, while simplifying, can be unrealistic in many scenarios. In the continual reinforcement learning scenario, the sequence of tasks is another source of nonstationarity. In this work, we propose to examine this continual reinforcement learning setting through the block contextual MDP (BC-MDP) framework, which enables us to relax the assumption of stationarity. This framework challenges RL algorithms to handle both nonstationarity and rich observation settings and, by additionally leveraging smoothness properties, enables us to study generalization bounds for this setting. Finally, we take inspiration from adaptive control to propose a novel algorithm that addresses the challenges introduced by this more realistic BC-MDP setting, allows for zero-shot adaptation at evaluation time, and achieves strong performance on several nonstationary environments.

One-sentence Summary: We introduce the Lipschitz Block Contextual MDP framework for the continual RL setting and propose a representation learning algorithm that enables RL agents to generalize to non-stationary environments.

Community Implementations: [ 6 code implementations](https://www.catalyzex.com/paper/arxiv:2110.06972/code)