Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model

Arie Wortsman Zurich, Hugo Tabanelli, Yatin Dandi, Florent Krzakala, Bruno Loureiro

Keywords: Spectral learning, High-dimensional, teacher-student, multi-index models, Deep versus shallow, learning theory, Gaussian Equivalence, Spectral methods, Power Laws, Scaling Laws

TL;DR: We propose a simple mechanism by which scaling laws emerge from feature learning in multi-layer networks in a mathematically solvable model

Abstract: We propose a simple mechanism by which scaling laws emerge from feature learning in multi-layer networks. We study a high-dimensional hierarchical target that is a globally high-degree function, but that can be represented by a combination of latent compositional features whose weights decrease as a power law. We show a layer-wise spectral algorithm exploiting this hierarchy recovers the latent directions sequentially: strong features become detectable at small sample sizes, while weaker features require more data. We prove sharp feature-wise recovery thresholds and show that aggregating these transitions yields an explicit power-law decay of the prediction error. Technically, the analysis relies on random matrix methods and a resolvent-based perturbation argument, which gives matching upper and lower bounds for individual eigenvector recovery beyond what standard gap-based perturbation bounds provide. Numerical experiments confirm the predicted sequential recovery, finite-size smoothing of the thresholds, and separation from non-hierarchical kernel baselines. Together, these results show how smooth scaling laws can emerge from a cascade of sharp feature-learning transitions.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 81