DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delays

Bo Xia, Yilun Kong, Bo Yuan, Yongzhe Chang, Zhiheng Li, Xueqian Wang

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Reinforcement learning, Random delay, Encoder, Interpretability

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Classic reinforcement learning (RL) frequently struggles with tasks involving delays due to the violation of the Markov property.
Existing approaches usually tackle this issue with end-to-end solutions using state augmentation, often by augmenting the state space with a predefined maximum dimension to accommodate random delays.
However, this black-box approach, characterized by incomprehensible intermediate processes and redundant information in augmented states, can result in instability and even undermine the overall performance.
To alleviate the delay challenges in RL, we propose DEER (Delay-resilient Encoder-Enhanced RL), a framework that can effectively enhance the interpretability and address the random delay issues.
DEER employs a pretrained encoder to encode delayed states along with their variable-length past action sequences due to different delays.
Specifically, we leverage delay-free environment datasets to train the encoder and convert delayed states and their corresponding action sequences into hidden states, which serve as novel delay-free states for further policy training.
In a variety of delayed scenarios, the trained encoder can smoothly integrate with standard RL algorithms without extra modifications and enhance the delay-solving capability by simply adapting the input dimension of the original algorithms.
We evaluate DEER through extensive experiments on Gym and Mujoco, which confirm that DEER is superior to state-of-the-art RL algorithms in both constant and random delay environments.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1630