State-Action Joint Regularized Implicit Policy for Offline Reinforcement Learning

Shentao Yang, Zhendong Wang, Huangjie Zheng, Mingyuan Zhou

Keywords: Implicit Policy, State-action Visitation, Distribution Matching, Generative Adversarial Networks

Abstract: Offline reinforcement learning enables learning from a fixed dataset, without further interactions with the environment. The lack of environmental interactions makes the policy training vulnerable to state-action pairs far from the training dataset and prone to missing rewarding actions. For training more effective agents, we propose a framework that supports learning a flexible and well-regularized policy, which consists of a fully implicit policy and a regularization through the state-action visitation frequency induced by the current policy and that induced by the data-collecting behavior policy. We theoretically show the equivalence between policy-matching and state-action-visitation matching, and thus the compatibility of many prior work with our framework. An effective instantiation of our framework through the GAN structure is provided, together with some techniques to explicitly smooth the state-action mapping for robust generalization beyond the static dataset. Extensive experiments and ablation study on the D4RL dataset validate our framework and the effectiveness of our algorithmic designs.

One-sentence Summary: An offline reinforcement learning framework that supports the learning of a flexible and well-regularized policy.