Generalizing the Geometry of Model Merging Through Frechet Averages

Marvin F. da Silva, Mohammed Adnan, Felix Dangel, Sageev Oore

Abstract:

Model merging aims to combine multiple mod-
els into one without additional training. Na¨ıve
parameter-space averaging can be fragile under
architectural symmetries, as their geometry does
not take them into account. In this work we show
that not only the geometry, but also the averaging
procedure itself, must be symmetry-invariant to
achieve symmetry-aware merges. Consequently,
we propose a general solution: merging as Fr´echet
averaging, i.e. selecting parameters that minimize
a sum of geodesic distances on an appropriate
manifold. In this view, the key design choice is
the overall geometry, i.e. , the choice of metric,
manifold, and distance approximation, that deter-
mines what it means for two models to be “close.”
We show that Fr´echet averaging, combined with
simplifying assumptions, contains Fisher merging.
Building on this, we examine the particular case
of low-rank adapters (LoRA), whose symmetries
induce a distinct geometry: that of a quotient man-
ifold. We outline the limitations of current LoRA
merging methods, propose a practical algorithm
for this setting, and show how they compare with
other commonly used approaches.