Time Limits in Reinforcement Learning

Fabio Pardo, Arash Tavakoli, Vitaly Levdik, Petar Kormushev

Abstract: In reinforcement learning, it is common to let an agent interact with its environment for a fixed amount of time before resetting the environment and repeating the process in a series of episodes. The task that the agent has to learn can either be to maximize its performance over (i) that fixed amount of time, or (ii) an indefinite period where the time limit is only used during training. In this paper, we investigate theoretically how time limits could effectively be handled in each of the two cases. In the first one, we argue that the terminations due to time limits are in fact part of the environment, and propose to include a notion of the remaining time as part of the agent's input. In the second case, the time limits are not part of the environment and are only used to facilitate learning. We argue that such terminations should not be treated as environmental ones and propose a method, specific to value-based algorithms, that incorporates this insight by continuing to bootstrap at the end of each partial episode. To illustrate the significance of our proposals, we perform several experiments on a range of environments from simple few-state transition graphs to complex control tasks, including novel and standard benchmark domains. Our results show that the proposed methods improve the performance and stability of existing reinforcement learning algorithms.

TL;DR: We consider the problem of learning optimal policies in time-limited and time-unlimited domains using time-limited interactions.

Keywords: reinforcement learning, Markov decision processes, deep learning

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:1712.00378/code)