Model Merging by Gradient Matching

Published: 02 Nov 2023, Last Modified: 18 Dec 2023UniReps PosterEveryoneRevisionsBibTeX
Supplementary Material: pdf
Keywords: Model Merging, Language Models, Gradient Matching
TL;DR: We connect model merging and gradient matching and show how reduced mismatch leads to better performance.
Abstract: Models trained on different datasets can be merged by a weighted-averaging of their parameters, but why does it work and when can it fail? Here, we connect the inaccuracy of weighted-averaging to mismatches in the gradients and propose a new uncertainty-based scheme to improve the performance by reducing the mismatch. The connection also reveals implicit assumptions in other schemes such as averaging, task arithmetic, and Fisher-weighted averaging.
Track: Extended Abstract Track
Submission Number: 55