Robust Weight Imprinting: Insights from Neural Collapse and Proxy-Based Aggregation

TMLR Paper5360 Authors

Abstract: The capacity of foundation models allows for their application to new, unseen tasks. The adaptation to such tasks is called transfer learning. An efficient transfer learning method that circumvents parameter optimization is imprinting. It has been reinvented several times, but not systematically studied. In this work, we propose the general $\texttt{IMPRINT}$ framework, identifying three main components: generation, normalization, and aggregation. Through the lens of this framework, we conduct an in-depth analysis and a comparison of the existing methods. Our findings reveal the benefits of representing novel data with multiple proxies in the generation step and show the importance of proper normalization. Beyond an extensive analytical grounding, our framework enables us to propose a novel variant of imprinting which outperforms previous work on transfer learning tasks by $6\%$. This variant determines proxies through clustering motivated by the neural collapse phenomenon -- a connection that we draw for the first time. We publicly release our code at (link removed for review).

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Steffen_Schneider1

Submission Number: 5360