Reinforcement Learning via Lazy-Agent for Environments with Random Delays

Jongsoo Lee, Jangwon Kim, Jiseok Jeong, Soohee Han

Keywords: Delayed system, Markov decision process, reinforcement learning

TL;DR: We introduce a simple yet effective reinforcement learning agent, referred to as the lazy-agent, which can be applied in environments with random delays.

Abstract: Real-world reinforcement learning applications are often hampered by delayed feedback from environments, which violates the fundamental assumption of the Markovian property and introduces significant challenges. While numerous methods have been proposed for handling environments with constant delays, those with random delays remain largely unexplored owing to their inherent complexity and variability. In this study, we explored environments with random delays and proposed a novel strategy to transform them into their equivalent constant-delay counterparts by introducing a simple agent called the *lazy-agent*. This approach naturally overcomes the challenges posed by the variability of random delays, enabling the application of state-of-the-art methods, originally designed for handling constant delays, to random-delay environments without any modification. Empirical results demonstrate that our lazy-agent significantly outperformed other baseline algorithms in terms of asymptotic performance and sample efficiency in random-delay environments.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2786