Provable Target Sample Complexity Improvements as Pre‑Trained Models Scale

Kazuto Fukuchi, Ryuichiro Hataya, Kota Matsui

Abstract: Pre-trained models have become indispensable for efficiently building models across a broad spectrum of downstream tasks. The advantages of pre-trained models have been highlighted by empirical studies on scaling laws, which demonstrate that larger pre-trained models can significantly reduce the sample complexity of downstream learning. However, existing theoretical investigations of pre-trained models lack the capability to explain this phenomenon. In this paper, we provide a theoretical investigation by introducing a novel framework, caulking, inspired by parameter-efficient fine-tuning (PEFT) methods such as adapter-based fine-tuning, low-rank adaptation, and partial fine-tuning. Our analysis establishes that improved pre-trained models provably decrease the sample complexity of downstream tasks, thereby offering theoretical justification for the empirically observed scaling laws relating pre-trained model size to downstream performance, a relationship not covered by existing results.

Code Dataset Promise: No

Code Dataset Url: https://github.com/moskomule/calking_aistats26

Format Confirmation: I agree that I have read and followed the formatting instructions for the camera ready version.

Submission Number: 728