Mutual Information Dynamics Learning: A New Paradigm for Unsupervised Reinforcement Learning

Yucheng Yang, Tianyi Zhou, Mykola Pechenizkiy, Meng Fang

Keywords: Unsupervised, Reinforcement Learning, Mutual Information Skill Learning

TL;DR: A novel unsupervised RL framework that leverage existing mutual information skill learning techniques to learn a mixture of diverse and transferable dynamic models.

Abstract: Unsupervised reinforcement learning (URL) aims to develop general-purpose agents that can adapt to unseen downstream tasks without relying on task-specific supervision. Existing approaches predominantly focus on learning diverse skills by maximizing mutual information, but they are often limited to simple navigation tasks and fail to scale to more complex domains such as robotic manipulation, where prior knowledge is typically required. In this work, we demonstrate that mutual information-based objectives can be leveraged far beyond skill learning. We propose a novel URL framework that trains exploratory skills to collect diverse transition data with distinct dynamics. This diverse dataset enables the training of a mixture of dynamic models, where each model captures the dynamics of a specific region. Collectively, these models provide comprehensive coverage of the dynamics required for a wide range of downstream tasks. Our straightforward and prior-free learning objective outperforms existing state-of-the-art skill discovery approaches in URL. Our results advocate a paradigm shift in URL, from skill learning toward dynamics learning, to acquire fully generalizable knowledge during pretraining.

Primary Area: reinforcement learning

Submission Number: 24322