Towards a Universal Scaling Law of LLM Training and Inference

Chuhan Wu, Ruiming Tang

Abstract: Guided by the prophecy of scaling law, large language models (LLMs) demonstrate higher levels of intelligence with increased sizes and computational power. Meanwhile , the overall outcome of small LLMs seems to show a scaling trend when a higher inference cost is paid in prompting and sampling. However, the inherent relatedness between training and inference in the path of scaling up is less studied. In this article, we present a universal theory on the joint computational scaling of LLM training and inference, which characterizes the general behaviors of LLM in various settings. Based on simple modeling of several key hyperparameters, we give intuitive explanations for the effectiveness of various techniques at both training and inference time. To explain the limitation of the current inference paradigm, we further propose the concept of meta-scaling to address the problem of error accumulation in the inference scaling process. We hope that this work can provide insight into future LLM research, development, and applications.