Linear Maps for Cross-Model Finetuning Transfer

Darrin O'Brien, Dhikshith Gajulapalli, Pranay Rishi Nalem, Alexander Ramsey, Eric Xia

Track: Extended Abstract Track

Keywords: model interpretability, transfer learning, model compression, knowledge transfer, low-resource learning, multi-task learning

TL;DR: We show that linear mappings from base to fine-tuned model representations -- task matrices -- can transfer fine-tuned capabilities to base models on diverse vision and text tasks.

Abstract:

Results in interpretability suggest that large vision and language models develop
implicit linearities in pretrained settings. Learned linear encodings have been
documented in in-context learning settings, where model predictions are biased
at runtime. However, it is unclear whether similar linear representations exists in
more generalized adaptation regimes. We investigate the existence of similar linear transformations from base to
finetuned embedding states. We demonstrate that for CLIP, DEiT, DINOv3, allMiniLM-V2, and RoBERTa, a base model
augmented with a task-specific matrix approaches finetuned accuracies on certain datasets,
while resulting in marginal improvements on others. Our results demonstrate
that over a range of models, modalities, and tasks, linear encoding in transformer
embedding spaces exist not only between layers in a single model architecture, but
also between pretrained and finetuned architectures.

Submission Number: 44