Scale-Equivariant Alignment: Closing the Residual Barrier After Permutation Matching

Kaustubh S. Bukkapatnam, Siddharth Karuturi

Keywords: Scale symmetry, Neural network merging, Permutation alignment

TL;DR: SEA jointly resolves permutation and scale symmetries in ReLU networks, reducing residual loss barriers by 74% without test-time overhead.

Abstract: Recent work shows that aligning permutation symmetries between two independently trained networks largely eliminates the loss barrier on linear interpolations (Ainsworth et al., 2023). A residual barrier persists, however, which Jordan et al. (2023) attribute to variance collapse — a phenomenon we show is a direct consequence of unresolved scale symmetries. ReLU networks possess a layerwise symmetry group that combines neuron permutations with positive rescalings, but existing alignment methods account only for permutations. We formally decompose the loss barrier into a permutation component and a scale component, prove a lower bound on the scale component in terms of mismatches in neuron scales between networks, and show that the optimal scale alignment is given by the geometric mean of per-neuron weight norms. Building on this, we propose Scale-Equivariant Alignment (SEA), an alternating algorithm that jointly solves for permutations and scales, and prove it converges monotonically to a critical point of a joint alignment objective. On CIFAR-10 ResNets, SEA reduces the residual loss barrier after permutation alignment by 74% and matches the accuracy of REPAIR without any test-time renormalization overhead.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 62