Parameter-Efficient Transfer Learning with Diff Pruning

Demi Guo, Alexander M Rush, Yoon Kim

Keywords: transfer learning, parameter efficiency

Abstract: While task-specific finetuning of deep networks pretrained with self-supervision has led to significant empirical advances in NLP, their large size makes the standard finetuning approach difficult to apply to multi-task, memory-constrained settings, as storing the full model parameters for each task become prohibitively expensive. We propose $\textit{diff pruning}$ as a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. This approach views finetuning as learning a task-specific diff vector that is applied on top of the pretrained parameter vector, which remains fixed and is shared across different tasks. The diff vector is adaptively pruned during training with a differentiable approximation to the $L_0$-norm penalty to encourage sparsity. Diff pruning becomes parameter-efficient as the number of tasks increases, as it requires storing only the nonzero positions and weights of the diff vector for each task, while the cost of storing the shared pretrained model remains constant. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark while only modifying 0.5$\%$ of the pretrained model's parameters per task.

One-sentence Summary: Parameter-efficient transfer learning for NLP by learning to finetune only a few parameters for each task

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2012.07463/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=aS-Jq68Dd5