Decoupled Prioritized Resampling: Advancing Offline RL with Improved Behavior Policy

Yang Yue, Bingyi Kang, Xiao Ma, Gao Huang, Shiji Song, Shuicheng YAN

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Offline reinforcement learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: This paper introduces a novel method enhancing offline RL by addressing the suboptimal policy constraint issue through specialized priority functions and unique decoupled resampling.

Abstract:

Offline reinforcement learning (RL) is challenged by the distributional shift problem.
To tackle this issue, existing works mainly focus on designing sophisticated policy constraints between the learned policy and the behavior policy.
However, these constraints are applied equally to well-performing and inferior actions through uniform sampling, which might negatively affect the learned policy.
In this paper, we proposeOffline Decoupled Prioritized Resampling (ODPR), which designs specialized priority functions for the suboptimal policy constraint issue in offline RL, and employs unique decoupled resampling for training stability.
Through theoretical analysis, we show that the distinctive priority functions induces a provable improved behavior policy by modifying the distribution of the original behavior policy, and when constrained to this improved policy, a policy-constrained offline RL algorithm is likely to yield a better solution.
We provide two practical implementations to balance computation and performance: one estimates priorities based on a fitted value network (ODPR-A), and the other utilizes trajectory returns (ODPR-R) for quick computation. ODPR serves as a highly compatible plug-and-play component with prevalent offline RL algorithms. We assess ODPR using five algorithms, namely BC, TD3+BC, Onestep RL, CQL, and IQL. Comprehensive experiments substantiate that both ODPR-A and ODPR-R markedly enhance the performance across all baseline methods.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 81