DPPA: Merging Large Language Model using Dynamic Pruning and Partition Amplification
Abstract: Model merging aims to combine models with different capabilities into a single unified model, providing multiple capabilities without the necessity of retraining with the original training data. However, as distinctions between fine-tuned and base models grow, especially for large language models, current methods suffer significant performance drops, hindering true multi-domain capabilities. In this study, we propose a two-stage method, called Dynamic Pruning and Partition Amplification (DPPA), to address the challenge of merging models with significant distinctions. First, we introduce Dynamic Pruning (DP) to discover significant parameters and remove redundant ones. Subsequently, we propose Dynamic Partition Amplification (DPA) to restore the capability in the domain. Experimental results demonstrate that our approach performs outstandingly, improving model merging performance by almost 20\%.
Paper Type: Long
Research Area: Efficient/Low-Resource Methods for NLP
Research Area Keywords: distillation
Languages Studied: english
Submission Number: 4886
Loading