Reinforcement Learning with Random Delays

Yann Bouteiller, Simon Ramstedt, Giovanni Beltrame, Christopher Pal, Jonathan Binas

Keywords: Reinforcement Learning, Deep Reinforcement Learning

Abstract: Action and observation delays commonly occur in many Reinforcement Learning applications, such as remote control scenarios. We study the anatomy of randomly delayed environments, and show that partially resampling trajectory fragments in hindsight allows for off-policy multi-step value estimation. We apply this principle to derive Delay-Correcting Actor-Critic (DCAC), an algorithm based on Soft Actor-Critic with significantly better performance in environments with delays. This is shown theoretically and also demonstrated practically on a delay-augmented version of the MuJoCo continuous control benchmark.

One-sentence Summary: We propose a framework for Reinforcement Learning with random action and observation delays.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Code: [ rmst/rlrd](https://github.com/rmst/rlrd) + [ 2 community implementations](https://paperswithcode.com/paper/?openreview=QFYnKlBJYR)

Data: [MuJoCo](https://paperswithcode.com/dataset/mujoco)

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2010.02966/code)