An offline-to-online reinforcement learning approach based on multi-action evaluation with policy extension

Xuebo Cheng, Xiaohui Huang, Zhichao Huang, Nan Jiang

Abstract: Offline Reinforcement Learning (Offline RL) is able to learn from pre-collected offline data without real-time interaction with the environment by policy regularization via distributional constraints or support set constraints. However, since the policy learned from offline data under the constrains of support set is usually similar to the behavioral policy due to the overly conservative constraints, offline RL confronts challenges in active behavioral exploration. Moreover, without online interaction, policy evaluation becomes prone to inaccuracy, and the learned policy may lack robustness in the presence of sub-optimal state-action pairs or noise in a dataset. In this paper, we propose an Offline-to-Online Reinforcement Learning Approach based on Multi-action Evaluation with Policy Extension(MAERL) for improving the ability of the policy exploration and the effective value evaluation of state-action in offline RL. In MAERL, we develop four modules: (1) in the policy extension module, we design a policy extension method, which uses the online policy to extend the offline policy; (2) in the multi-action evaluation module, we present an adaptive manner to merge the offline and online policies to generate an action of the agent; (3) in the action-oriented module, we learn the action trajectories of the agent from the dataset, mitigating the issue of actions deviating excessively during environmental exploration; (4) to maintain the consistency in the agent’s actions, we propose an action temporally-aligned representation learning method to maintain the trend of actions of agents. This approach ensures that the agent’s actions align with the learned trajectories, preventing significant deviations during exploration. Extensive experiments are conducted on 15 scenarios of the D4RL/mujoco environment. Results demonstrate that our proposed methods achieve the best performance in 12 scenarios and the second-best performance in 3 scenarios compared to state-of-the-art methods. The project’s code can be found at https://github.com/FrankGod111/Policy-Expansion.git