Unified Neural Scaling Laws

Ethan Caballero, Priyank Jaini, David Krueger, Irina Rish

Keywords: scaling laws, scaling law, neural scaling laws, neural scaling law, broken neural scaling laws, broken neural scaling law

TL;DR: Multivariate Neural Scaling Laws

Abstract: We present a functional form (that we refer to as a Unified Neural Scaling Law (UNSL)) that accurately models and extrapolates the scaling behaviors of deep neural networks as multiple dimensions all vary simultaneously (i.e. how the evaluation metric of interest varies as one simultaneously varies the number of model parameters, training dataset size, number of training steps, and various hyperparameters) for various architectures and for each of various tasks within a varied set of upstream and downstream tasks. When compared to other functional forms for neural scaling, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 23924