Realistic Evaluation of Model Merging for Compositional Generalization

Derek Tam, Yash Kant, Brian Lester, Igor Gilitschenski, Colin Raffel

Keywords: model merging, realistic evaluation, compositional generalization

TL;DR: A realistic evaluation of model merging methods testing compositional generalization in computer vision and natural language processing.

Abstract:

Merging has become a widespread way to cheaply combine individual models into a single model that inherits their capabilities and attains better performance.
This popularity has spurred rapid development of many new merging methods, which are typically validated in disparate experimental settings and frequently differ in the assumptions made about model architecture, data availability, and computational budget.
In this work, we characterize the relative merits of different merging methods by evaluating them in a shared experimental setting and precisely identifying the practical requirements of each method.
Specifically, our setting focuses on using merging for $\textit{compositional generalization}$ of capabilities in image classification, image generation, and natural language processing.
Additionally, we measure the computational costs of different merging methods as well as how they perform when scaling the number of models being merged.
Taken together, our results clarify the state of the field of model merging and provide a comprehensive and rigorous experimental setup to test new methods.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8094