Exploring Model Kinship for Merging Large Language Models

ACL ARR 2025 May Submission1098 Authors

Abstract: Model merging has become one of the key technologies for enhancing the capabilities and efficiency of Large Language Models (LLMs). The open-source community has driven model evolution by iteratively merging existing models. However, a principled understanding of the expected gains and underlying factors in model merging remains lacking. In this work, we examine model evolution through continual merging, analogous to biological evolution, and introduce the concept of model kinship, the degree of similarity or relatedness between LLMs. With comprehensive empirical analysis, we find that there is a certain relationship between model kinship and the performance gains after model merging, which can help guide our selection of candidate models. Inspired by this, we propose a new model merging strategy: Top-$k$ Greedy Merging with Model Kinship, which can yield better performance on benchmark datasets. Specifically, we discover that using model kinship as a criterion can assist us in continuously performing model merging, alleviating the degradation (local optima) in model evolution, whereas model kinship can serve as a guide to escape these traps.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: model merging, model kinship, large language models

Contribution Types: Approaches low compute settings-efficiency

Languages Studied: English

Keywords: model merging, model kinship, large language models

Submission Number: 1098