Meta-Adapter: Parameter Efficient Few-Shot Learning through Meta-Learning

Anonymous

Abstract: With consistent improvements in the representational capacity of large pre-trained transformers, it has become increasingly viable to serve these models as shared backbones that enable modeling a large number of tasks simultaneously. However, fine-tuning the entire model for every task of interest makes a copy of all the model parameters, rendering such scenarios highly impractical. Recently introduced Adapter methods propose a promising alternative, one where only a small number of additional parameters are introduced per task specifically for fine-tuning. However, Adapter often require large amounts of task-specific data for good performance and don't work well in data-scarce few-shot scenarios. In this paper, we take a meta-learning viewpoint for parameter-efficient fine-tuning in few-shot settings. We introduce Meta-Adapter, which are small blocks of meta-learned adapter layers inserted in a pre-trained model that re-purpose a frozen pre-trained model into a parameter-efficient few-shot learner. Meta-Adapter perform competitively with state-of-the-art few-shot learning methods, that require full fine-tuning, while only fine-tuning 0.6\% of the parameters. We evaluate Meta-Adapter along with multiple transfer learning baselines on an evaluation suite of 17 classification tasks and find that they improve few-shot learning accuracy by a large margin over competitive parameter-efficient methods while requiring significantly lesser parameters for fine-tuning.