Scaling Parameter-Efficiency with Distribution Shifts for Domain Adaptation

Samuel Osebe, Jay Taneja

Keywords: peft, remote-sensing

Abstract: Distribution shifts between source and target domains pose significant challenges to the generalization capabilities of machine learning models. While foundation models are often fine-tuned to adapt to new domains, their increasing size has led to a rise in the computational resources required for domain adaptation. This has driven interest in Parameter-Efficient Fine-Tuning (PEFT) methods, which have shown strong performance on in-domain tasks. In this work, we investigate how PEFT methods scale with varying degrees of distribution shifts and propose a novel PEFT method designed for domain adaptation. We select an English pre-trained Large Language Model (LLM) as the foundation model and apply PEFT techniques across tasks that progressively introduce larger distribution shifts. Specifically, we begin with SuperGLUE English benchmark, followed by a multilingual inference task for high-resource and low-resource languages, then a multimodal image captioning task. Finally, We introduce a novel multimodal and multitemporal radar interferometry task for detecting charcoal production sites in remote areas. Separately, we propose a PEFT method that augments matrix vector products with learnable parameters, inducing a learning paradigm that conditions on both training data and encoded information. Our method is competitive against SOTA PEFT methods for English tasks and out-performs SOTA methods for larger distribution shifts i.e. low-resource multilingual, image captioning, and radar interferometry tasks.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 10053