Interpolated Layer-Wise Merging for NeurIPS 2024 LLM Merging Competition

Rio Akizuki, Nozomu Yoshinari, Yuya Kudo, Yoichi Hirose, Toshiyuki Nishimoto, Kento Uchida, Shinichi Shirakawa

Keywords: model merging, large language models

Abstract: Model merging is an emerging technique to generate models by combining multiple models, leveraging their individual strengths in a unified model. Because the configuration hyperparameters of model merge techniques have a significant effect on the merged models, optimizing them using evolutionary algorithms is promising to enhance the model merge results. In our participation in the LLM Merging Competition in NeurIPS 2024, we introduced a novel search space, interpolated layer-wise space, for optimizing merging configurations using evolutionary algorithms. This paper explores the potential for performance enhancement in merged models and the efficiency of our search space for evolutionary model merge.

Submission Number: 7