Adaptive Reinforcement Learning for Unobservable Random Delays

ICLR 2026 Conference Submission16391 Authors

Keywords: Reinforcement learning, Delay, Unobservability

TL;DR: We introduce a novel framework enabling RL agents to adapt on-the-fly to unobservable random delays.

Abstract: In standard Reinforcement Learning (RL) settings, the interaction between the agent and the environment is typically modeled as a Markov Decision Process (MDP), which assumes that the agent observes the system state instantaneously, selects an action without delay, and executes it immediately. In real-world dynamic environments, such as cyber-physical systems, this assumption often breaks down due to delays in the interaction between the agent and the system. These delays can vary stochastically over time and are typically _unobservable_ when deciding on an action. Existing methods deal with this uncertainty conservatively by assuming a known fixed upper bound on the delay, even if the delay is often much lower. In this work, we introduce the _interaction layer_, a general framework that enables agents to adaptively handle unobservable and time-varying delays. Specifically, the agent generates a matrix of possible future actions, anticipating a horizon of potential delays, to handle both unpredictable delays and lost action packets sent over networks. Building on this framework, we develop a model-based algorithm, _Actor-Critic with Delay Adaptation (ACDA)_, which dynamically adjusts to delay patterns. Our method significantly outperforms state-of-the-art approaches across a wide range of locomotion benchmark environments.

Primary Area: reinforcement learning

Submission Number: 16391