DPPA: Pruning Method for Large Language Model to Model Merging

Anonymous

Abstract: Model merging is to combine fine-tuned models from multiple domains to enhance the model's capabilities across various domains. Merging performance degradation is due to parameter conflicts. The prevailing methods address this issue of parameter conflicts during the merging stage, but recently scholars have been paying more attention to resolving this problem during the pruning stage. DARE has demonstrated promising results on a simple fine-tuned model. However, this approach exhibit diminished effectiveness when applied to complex fine-tuned models that has significant parameter bias compared to the baseline model. In this study, we propose a two-stage method called DPPA to address the challenge of fusing complex fine-tuned models. First, we introduce Dynamically Pruning (DP), an improved approach based on magnitude pruning which aim is to enhance performance at higher pruning rates. Subsequently, we propose Dynamically Partition Amplification (DPA), a rescaling technique that aims to dynamically amplify partitions of parameters based on their varying levels of significance. The experimental results show that our approach retains only 20\% of the specific domain parameters, yet achieves comparable performance to other methods that retain 90\% of the specific domain parameters. Furthermore, our method, due to its exceptional performance after pruning, also achieves a significant improvement of nearly 20\% in model merging. We will make our code on Github

Paper Type: long

Research Area: Efficient/Low-Resource Methods for NLP

Languages Studied: English