Generalizability of Mixture of Out-of-Domain Adapters from the Lens of Signed Weight Directions and its Application to Effective Model Pruning

Anonymous

Abstract: Several parameter-efficient fine-tuning methods based on adapters have been introduced as a streamlined approach to incorporate not only a single specialized knowledge into existing Large Language Models (LLMs) but also multiple of them at once. However, understanding their generalizability across different out-of-domain tasks and their adversarial robustness remains unexplored. Thus, in this study, we conduct a comprehensive analysis to elucidate the workings of the Mixture of Out-of-Domain Adapters, offering insights across various facets, ranging from training data characteristics to the intricacies of adapter weights within the framework of the Mixture of Adapters. Specifically, we propose to analyze how the signed directions of adapters' weights during mixing correlate with their generalizability and how such analysis allows us to design a more effective model pruning algorithm that balances space, time, and predictive performance. The source code of the paper will be publicly available.

Paper Type: long

Research Area: Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability, Reproduction study

Languages Studied: English