Understanding Feature Learning Dynamics in Isotropic Regularizers via BHEP Statistics

Jeongyou Lee, Sungyoon Lee

Keywords: Self-Supervised Learning, Learning Dynamics, JEPA, LeJEPA, SIGReg, Regularizer, Embedding Distribution

TL;DR: We utilize the BHEP statistics to analyze macroscopics dynamics of SIGReg

Abstract: Self-supervised training of LeJEPA is governed by the SIGReg regularizer to ensure an isotropic Gaussian embedding distribution and prevent dimensional collapse. However, directly analyzing its learning dynamics remains mathematically challenging, due to its random projection and lack of closed-form expressions. In this work, we bridge this gap by reformulating the SIGReg objective into a tractable BHEP-type statistic that shares the identical target distribution. Leveraging this theoretical proxy, we derive exact closed-form ODEs governing the macroscopic eigenvalue learning dynamics. Based on this formulation, we characterize the feature learning dynamics through two distinct phase transitions: a rapid initial $\textit{Explosion Phase}$ followed by a self-stabilizing $\textit{Shrinking Phase}$. Crucially, we decouple the global driving force into a collapsing $\textit{engine}$ and a stiffening $\textit{brake}$. Our exact analytic analysis reveals an extreme asymmetric decay between these forces, mathematically proving how the early saturation of top features profoundly depletes the driving force for subsequent features. This mechanism explains the severe, non-linear stepwise delay observed during training, while ultimately guaranteeing stable convergence toward isotropy.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 145