Representation Convergence: Mutual Distillation is Secretly a Form of Regularization

Zhengpeng Xie, Jiahang Cao, Changwei Wang, Fan Yang, Marco Hutter, Qiang Zhang, Jianxiong Zhang, Renjing Xu

Keywords: Online Reinforcement Learning, Generalization Theory, Representation Learning, Knowledge Distillation

TL;DR: A novel theoretical framework for formalizing generalization in reinforcement learning.

Abstract: In this paper, we argue that mutual distillation between reinforcement learning policies serves as an implicit regularization, preventing them from overfitting to irrelevant features. We highlight two separate contributions: (i) Theoretically, for the first time, we provide an end-to-end theoretical proof that enhancing the policy robustness to irrelevant features leads to improved generalization performance. (ii) Empirically, we demonstrate that mutual distillation between policies contributes to such robustness, enabling the spontaneous emergence of invariant representations over pixel inputs. Ultimately, we do not claim to achieve state-of-the-art performance but rather focus on uncovering the underlying principles of generalization and deepening our understanding of its mechanisms.

Primary Area: reinforcement learning

Submission Number: 4259