Features are fate: a theory of transfer learning in high-dimensional regression

Javan Tahir, Surya Ganguli, Grant M. Rotskoff

Abstract:

With the emergence of large-scale pre-trained neural networks, methods to adapt such "foundation" models to data-limited downstream tasks have become a necessity.
Fine-tuning, preference optimization, and transfer learning have all been successfully employed for these purposes when the target task closely resembles the source task, but a precise theoretical understanding of ``task similarity'' is still lacking.
We adopt a \emph{feature-centric} viewpoint on transfer learning and establish a number of theoretical results that demonstrate that when the target task is well represented by the feature space of the pre-trained model, transfer learning outperforms training from scratch.
We study deep linear networks as a minimal model of transfer learning in which we can analytically characterize the transferability phase diagram as a function of the target dataset size and the feature space overlap.
For this model, we establish rigorously that when the feature space overlap between the source and target tasks is sufficiently strong, both linear transfer and fine-tuning improve performance, especially in the low data limit.
These results build on an emerging understanding of feature learning dynamics in deep linear networks, and we demonstrate numerically that the rigorous results we derive for the linear case also apply to nonlinear networks.

Lay Summary:

This paper looks at how and why transfer learning works—especially when you don’t have much data to work with. Transfer learning is a popular machine learning technique where a model trained on one task is reused or adapted to help solve a different, often smaller or harder, task. A common belief is that if two tasks are similar enough—say, they have similar data or patterns—then a model trained on one should work well on the other. However, we show that this assumption doesn’t always hold. We argue that what really matters isn’t how similar the tasks look from the outside, but whether the original model has learned features that are actually useful for the new task.

To explore this, we use a simplified type of model called a "deep linear network," which is easier to analyze mathematically. This allows us to clearly see what makes transfer learning succeed or fail. Our key finding is that success depends on whether the new task is aligned with the kinds of patterns, or "features," the model has already learned. If the model’s internal knowledge includes the kinds of patterns needed for the new task, transfer learning can be very effective—even if the tasks don’t seem obviously similar. On the other hand, if the model hasn’t learned anything relevant to the new task, then transferring what it knows won’t help much, and might even hurt performance.

Overall, we suggest that instead of relying on surface-level similarities between datasets, we should focus more on whether the model's inner understanding is a good match for the new problem. This insight could help people make better choices when reusing machine learning models, especially in situations where collecting data is expensive or difficult.

Link To Code: https://github.com/javantahir/features_are_fate

Primary Area: Deep Learning->Theory

Keywords: transfer learning, feature learning, random matrix theory

Submission Number: 12448