Debugging ReBasin: What Limits Symmetry-Based Model Merging?

Elliot Stein, Christine Evers, Jonathon Hare

Keywords: Model merging, weight-space symmetries, linear mode connectivity, representation alignment, rebasin

TL;DR: We introduce representation-merge diagnostics that decompose symmetry-based model-merging failures, showing that narrow ResNets are limited mainly by residual-stream deployment constraints, while richer trained alignments help most at narrow width.

Abstract: Weights-merge accuracy is the standard measure of model-merging success, but it does not reveal whether failures come from poor alignment, architectural constraints, folding error, or normalisation. We introduce a representation-merge diagnostic that applies the same alignment maps directly at runtime, then progressively constrains this idealised merge to isolate contributors to the final weights-merge result. On CIFAR-100 with VGG-16-BN and ResNet-20, this decomposition shows that narrow-ResNet failures are dominated by a structural constraint: one residual-stream alignment must work at multiple positions within a stage. Trained encoder--decoder maps outperform closed-form alignments at narrow width, but this gap closes as width grows. Together, these results show that symmetry-based merging is limited both by the ability to find good alignments and by the ability of the architecture to realise them.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 46