Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass

Tong Chen, Hao Fang, Patrick Xia, Xiaodong Liu, Benjamin Van Durme, Luke Zettlemoyer, Jianfeng Gao, Hao Cheng

Keywords: language model; fine-tuning; prompting; adaptation

Abstract:

Large language models (LMs) acquire substantial knowledge during pretraining but often need adaptation to new contexts, tasks, or domains, typically achieved through fine-tuning or prompting. However, fine-tuning incurs significant training costs, while prompting increases inference overhead. Inspired by fast weight memory, we introduce GenerativeAdapter, an effective and efficient adaptation method that encode test-time context into LM's parameters with a single forward pass.
GenerativeAdapter augments a frozen pretrained LM with a lightweight adapter generator, trained via self-supervised learning, to produce parameter-efficient adapters.
Notably, our generator is general-purpose, i.e., one generator can adapt the corresponding base model for all langauge processing scenarios.
We apply GenerativeAdapter to two pretrained LMs (Mistral-7B and Llama2-7B) and evaluate the adapted models across knowledge acquisition from documents, learning from demonstrations, and personalization for users.
Overall, GenerativeAdapter provides a viable solution for adapting large LMs to evolving information and providing tailored user experience, while reducing training and inference costs relative to traditional fine-tuning and prompting techniques.

Submission Number: 125