Model Fusion of Heterogeneous Neural Networks via Cross-Layer Alignment

Dang Nguyen, Khai Nguyen, Nhat Ho, Dinh Phung, Hung Bui

Keywords: Model Fusion, Cross-layer Alignment, Knowledge Distillation, Model Compression, Model Transfer

Abstract: Layer-wise model fusion via optimal transport, named OTFusion, applies soft neuron association for unifying different pre-trained networks to save computational resources. While enjoying its success, OTFusion requires the input networks to have the same number of layers. To address this issue, we propose a novel model fusion framework, named CLAFusion, to fuse neural networks with a different number of layers, which we refer to as heterogeneous neural networks, via cross-layer alignment. The cross-layer alignment problem, which is an unbalanced assignment problem, can be solved efficiently using dynamic programming. Based on the cross-layer alignment, our framework balances the number of layers of neural networks before applying layer-wise model fusion. Our synthetic experiments indicate that the fused network from CLAFusion achieves a more favorable performance compared to the individual networks trained on heterogeneous data without the need for any retraining. With an extra finetuning process, it improves the accuracy of residual networks on the CIFAR10 dataset. Finally, we explore its application for model compression and knowledge distillation when applying to the teacher-student setting.

One-sentence Summary: A novel framework that can fuse neural networks with a different number of layers.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2110.15538/code)