A Free-Energy Principle for Representation Learning

Yansong Gao, Pratik Chaudhari

Keywords: information theory, thermodynamics, rate-distortion theory, transfer learning, information bottleneck

TL;DR: We use ideas from thermodynamics and information theory to design stochastic processes to keep the classification loss constant as the model is transferred from the source task to the target task.

Abstract: We employ a formal connection of machine learning with thermodynamics to characterize the quality of learnt representations for transfer learning. We discuss how information-theoretic functionals such as rate, distortion and classification loss of a model lie on a convex, so-called equilibrium surface. We prescribe dynamical processes to traverse this surface under constraints, e.g., an iso-classification process that trades off rate and distortion to keep the classification loss unchanged. We demonstrate how this process can be used for transferring representations from a source dataset to a target dataset while keeping the classification loss constant. Experimental validation of the theoretical results is provided on standard image-classification datasets.