SimMerge: Learning to Select Merge Operators from Similarity Signals

Oliver Bolton, Aakanksha, Arash Ahmadian, Sara Hooker, Marzieh Fadaee, Beyza Ermis

Abstract:

Model merging combines multiple models into a single model with aggregated capabilities, making it a powerful tool for large language model (LLM) development.
However, merge performance heavily depends on the choice of merge operator and merge order, often requiring expensive merge-and-evaluate searches to maximize.
In this work, we introduce SimMerge, a predictive merge-selection method that identifies high-performing merges using inexpensive similarity signals between models.
Given a small set of unlabeled probes, SimMerge extracts functional and structural features to predict the performance of candidate two-way merges, enabling merge operator and order selection without iterative evaluation.
We show that SimMerge consistently outperforms the best fixed merge operator across 7B-parameter LLMs and generalizes to multi-way merges and 111B-parameter LLMs without retraining. We further introduce a bandit variant that supports adding new tasks and operators online. Our results suggest that learning how to merge enables scalable model composition when checkpoint catalogs are large and evaluation budgets are limited.

Submission Number: 40