LLM Merging Competition Technical Report: Efficient Model Merging with Strategic Model Selection, Merging, and Hyperparameter Optimization

Zixiang Di, Yaoming Yang, Mei Jiang, Bingdong Li, Hong Qian, Aimin Zhou

Keywords: Large language model, model merging, multi-objective optimization

Abstract: The LLM Merging Competition in NeurIPS’24 aims to build LLMs efficiently through model merging, which enables the combination of multiple specialized fine-tuned models into a single model without the need for additional training. However, existing model merging techniques often suffer from performance degradation or require the model maker’s apriori knowledge or intuition to set hyperparameters. To address these challenges, this technical report proposes an efficient model merging (EMM) approach with three key modules: (1) strategic model selection, (2) hybrid merging algorithm, and (3) hyperparameter optimization. Our approach aims to improve the effectiveness of merged model while maintaining efficiency throughout the merging process. Experiments on different tasks verify the effectiveness of the proposed approach.

Submission Number: 4