DPPA: Pruning Method for Large Language Model to Model Merging

ACL ARR 2024 April Submission616 Authors

Abstract: Model merging is the process of combining models from various domains into a single model with multi-domain capabilities, and the challenge is to resolve parameter conflicts. To reduce the possibility of parameter conflicts, the pruning method is used to remove parameters from a model. The recent method utilizes a domain-independent pruning technique which is based on the assumption that there is little variation between different model parameters. We found that because domain-independent methods remove some domain-specific parameters, they are ineffective when there are significant distinctions in model parameters. In this paper, we address the challenge of merging models with significant distinctions by proposing a two-stage method called DPPA. First, we introduce Dynamically Pruning (DP) to discover domain-specific significant parameters and remove redundant ones. Subsequently, to enhance the capability in the domain, we propose Dynamical Partition Amplification (DPA), which amplifies significant parameters during the merging process. The results of the experiments demonstrate that our approach performs outstandingly, improving model merging performance by almost 20\%. We will share our code on GitHub.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: distillation

Languages Studied: English

Submission Number: 616