Connecting Independently Trained Modes via Layer-Wise Connectivity

Yongding Tian, Zaid Al-Ars, Maksim Kitsak, H Peter Hofstee

Keywords: Neural Network, Mode Connectivity

TL;DR: We propose LLPF algorithm, which leverages the layer-wise mode connectivity to reliably construct mode connectivity across a broader range of model families.

Abstract: Empirical and theoretical studies have shown that continuous low-loss paths can be constructed between independently trained neural network models. This phenomenon, known as mode connectivity, refers to the existence of such paths between distinct modes—i.e., well-trained solutions in parameter space. However, existing empirical methods are primarily effective for older and relatively simple architectures such as basic CNNs, VGG, and ResNet, raising concerns about their applicability to modern and structurally diverse models. In this work, we propose a new empirical algorithm for connecting independently trained modes that generalizes beyond traditional architectures and supports a broader range of networks, including MobileNet, ShuffleNet, EfficientNet, RegNet, Deep Layer Aggregation (DLA), and Compact Convolutional Transformers (CCT). In addition to broader applicability, the proposed method yields more consistent connectivity paths across independently trained mode pairs and supports connecting modes obtained with different training hyperparameters.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 10028