Provable Target Sample Complexity Improvements as Pre‑Trained Models Scale

Kazuto Fukuchi, Ryuichiro Hataya, Kota Matsui

Abstract: Pre-trained models have become indispensable for efficiently building models across a broad spectrum of downstream tasks. The advantage of the pre-trained model has been highlighted by empirical studies on scaling laws demonstrate that larger pre-trained models can significantly reduce the sample complexity of downstream learning. However, the existing theoretical investigations of the pre-trained model lack the capability in explaining such a phenomenon. In this paper, we provide a theoretical investigation of this phenomenon by introducing a novel framework, caulking, inspired by recent parameter-efficient fine-tuning (PEFT) methods, such as adapter-based fine-tuning, low-rank adaptation, and partial fine-tuning. Our analysis establishes that improved pre-trained models provably decrease the sample complexity of downstream tasks, thereby offering theoretical justification for the empirically observed scaling laws relating pre-trained model size to downstream performance, which is not covered by the existing results.

Submission Number: 728