Cascade of phase transitions in the training of energy-based models

Dimitrios Bachtis, Giulio Biroli, Aurélien Decelle, Beatriz Seoane

Keywords: Restricted Boltzmann Machine, Generative model, Phase transition, statistical physics, Energy-based model

TL;DR: We show theoretically and numerically that the training of Energy-based models undergoes several phase transitions.

Abstract: In this paper, we investigate the feature encoding process in a prototypical energy-based generative model, the Restricted Boltzmann Machine (RBM). We start with an analytical investigation using simplified architectures and data structures, and end with numerical analysis of real trainings on real datasets. Our study tracks the evolution of the model’s weight matrix through its singular value decomposition, revealing a series of thermodynamic phase transitions that shape the principal learning modes of the empirical probability distribution. We first describe this process analytically in several controlled setups that allow us to fully monitor the training dynamics until convergence. We then validate these findings by training the Bernoulli-Bernoulli RBM on real data sets. By studying the phase behavior over data sets of increasing dimension, we show that these phase transitions are genuine in the thermodynamic sense. Moreover, we propose a mean-field finite-size scaling hypothesis, confirming that the initial phase transition, reminiscent of the paramagnetic-to-ferromagnetic phase transition in mean-field ferromagnetism models, is governed by mean-field critical exponents.

Primary Area: Generative models

Submission Number: 6978