Towards Zero-Shot Generalization in Offline Reinforcement Learning
Keywords: offline reinforcement learning, generalization
Abstract: In this work, we study offline reinforcement learning (RL) with zero-shot generalization property (ZSG), where the agent has access to an offline dataset including experiences from different environments, and the goal of the agent is to train a policy over the training environments which performs well on test environments without further interaction. Existing work showed that classical offline RL fails to generalize to new, unseen environments. We propose pessimistic empirical risk minimization (PERM) and pessimistic proximal policy optimization (PPPO), which leverage pessimistic policy evaluation to guide policy learning and enhance generalization. Theoretically, our framework is capable of finding a near-optimal policy with ZSG. Empirically, our framework demonstrates the ability to enhance the performance of the base offline RL methods. Our result serves as a first step in understanding the foundation of the generalization phenomenon in offline reinforcement learning. Our codes are released at [this link](https://anonymous.4open.science/r/ProcgenExp-B5B4).
Primary Area: reinforcement learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 11426
Loading