Broken Neural Scaling Laws

Ethan Caballero, Kshitij Gupta, Irina Rish, David Krueger

Keywords: Scaling Laws, Scaling, Scale, Big Learning, Deep Learning, Artificial Neural Networks

TL;DR: We present a functional form that accurately models the scaling behaviors for various tasks from a very large and diverse set of downstream (and upstream) tasks, even scaling behaviors that were previously believed to be "unpredictable".

Abstract: We present a smoothly broken power law functional form that accurately models and extrapolates the scaling behaviors of deep neural networks (i.e. how the evaluation metric of interest varies as the amount of compute used for training, number of model parameters, training dataset size, or upstream performance varies) for various architectures and for various tasks within a large and diverse set of upstream and downstream tasks, in zero-shot, prompted, and fine-tuned settings. This set includes large-scale vision, language, audio, video, diffusion generative modeling, multimodal learning, contrastive learning, AI alignment, robotics, arithmetic, unsupervised/self-supervised learning, and reinforcement learning (single agent and multi-agent). When compared to other functional forms for neural scaling behavior, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set. Moreover, this functional form accurately models and extrapolates scaling behavior that other functional forms are incapable of expressing such as the non-monotonic transitions present in the scaling behavior of phenomena such as double descent and the delayed, sharp inflection points present in the scaling behavior of tasks such as arithmetic. Lastly, we use this functional form to glean insights about the limit of the predictability of scaling behavior. See https://arxiv.org/abs/2210.14891 for longer version of this paper. Code is available at https://github.com/ethancaballero/broken_neural_scaling_laws

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2210.14891/code)