Model-Based Transfer RL with Task-Agnostic Offline Pretraining

Minting Pan, Yitao Zheng, Haijian Chen, Yumeng He, Yunbo Wang, Xiaokang Yang

Keywords: World model, visual reinforcement learning, transfer learning

Abstract: Pretraining RL models on offline datasets is a promising way to improve their training efficiency in online tasks, but challenging due to the inherent mismatch in dynamics and behaviors across tasks or data domains. We present Vid2Act, a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from various offline datasets to a novel task. The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance for both dynamics representation transfer and policy transfer. Specifically, we build a time-varying, task-selective distillation loss to generate a set of offline-to-online similarity weights. These weights serve two purposes: (i) adaptively transferring the task-agnostic knowledge of physical dynamics to facilitate world model training, and (ii) learning to replay relevant source actions to guide the target policy. We demonstrate the advantages of Vid2Act over the state-of-the-art methods in Meta-World and DeepMind Control Suite.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5731