Toward Data Efficient Model Merging between Different Datasets without Performance Degradation

Masanori Yamada, Tomoya Yamashita, Shin'ya Yamaguchi, Daiki Chijiwa

Keywords: Model Merging, Mode Connectivity

Verify Author List: I have double-checked the author list and understand that additions and removals will not be allowed after the submission deadline.

TL;DR: We merge models trained on completely different tasks.

Abstract: Model merging is attracting attention as a novel method for creating a new model by combining the weights of different trained models. While previous studies reported that model merging works well for models trained on a single dataset with different random seeds, model merging between different datasets remains unsolved. In this paper, we attempt to reveal the difficulty in merging such models trained on different datasets and alleviate it. Our empirical analyses show that, in contrast to the single-dataset scenarios, dataset information needs to be accessed to achieve high accuracy when merging models trained on different datasets. However, the requirement to use full datasets not only incurs significant computational costs but also becomes a major limitation when integrating models developed and shared by others. To address this, we demonstrate that dataset reduction techniques, such as coreset selection and dataset condensation, effectively reduce the data requirement for model merging. In our experiments with SPLIT-CIFAR10 model merging, the accuracy is significantly improved by $31$\% when using the full dataset and $24$\% when using the sampled subset compared with not using the dataset. Our code is available at https://github.com/MasanoriYamada/re-basin-merge-pytorch.

Primary Area: Deep Learning (architectures, deep reinforcement learning, generative models, deep learning theory, etc.)

Paper Checklist Guidelines: I certify that all co-authors of this work have read and commit to adhering to the guidelines in Call for Papers.

Student Author: No

Submission Number: 89