A Geometric View of Model Merging: Quotient Fréchet Averages from Toy Models to LoRA

Marvin F. da Silva, Mohammed Adnan, Felix Dangel, Sageev Oore

Keywords: symmetry, geometry, merging, riemannian, lora

Abstract: Model merging combines task-specialized models without additional training, but weight-space averaging is not intrinsically well-defined when parameters have symmetries. Two checkpoints can represent the same or closely related function while occupying different points along a symmetry orbit; averaging arbitrary representatives can therefore produce a degraded merge. We formulate model merging as Fréchet averaging on a Riemannian parameter geometry, and argue that in the presence of symmetries the appropriate object is a Fréchet mean on the quotient space of parameterizations modulo symmetry. We work through a two-parameter linear model with a scaling symmetry, where the quotient geometry is closed form: quotient GeoMerge reduces to averaging the invariant predictors, while Euclidean and Fisher-weighted parameter averaging can collapse for gauge-equivalent checkpoints. We then instantiate the same principle for LoRA adapters and report proof-of-concept ViT-B/32 merging results, where quotient-aware GeoMerge improves over strong alignment-based LoRA merging baselines.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 69