Value Explicit Pretraining for Learning Transferable Representations
Keywords: Pretraining for RL, Contrastive Learning, Transfer learning for Robotics
Abstract: Understanding visual inputs for a given task amidst varied changes is a key challenge posed by visual reinforcement learning agents. We propose \textit{Value Explicit Pretraining} (VEP), a method that learns generalizable representations for transfer reinforcement learning. VEP enables efficient learning of new tasks that share similar objectives as previously learned tasks, by learning an objective-conditioned encoder that is invariant to changes in environment dynamics and appearance. To pre-train the encoder from a sequence of observations, we use a self-supervised contrastive loss that enables the model to relate states across different tasks based on the Bellman return estimate that is reflective of task progress, resulting in temporally smooth representations that capture the objective of the task. Experiments on a realistic navigation simulator and Atari benchmark show VEP outperforms current SoTA pretraining methods on the ability to generalize to unseen tasks. VEP achieves up to a $2\times$ improvement in rewards, and up to a $3\times$ improvement in sample efficiency. For videos of policy performance visit our \href{https://sites.google.com/view/value-explicit-pretraining/}{website}.
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 13087
Loading