RepFC: Universal Structural Reparametrization Block for High Performance, Lightweight Deep Neural Networks

Shambhavi Balamuthu Sampath, Judeson Anthony Fernando, Moritz Thoma, Nael Fasfous, Lukas Frickenstein, Pierpaolo Morì, Manoj Rohit Vemparala, Alexander Frickenstein, Ulf Schlichtmann, Walter Stechele

Abstract: Structural reparametrization (SRP) involves a bipartite network design, (1) an expanded multi-branch train-time network that is equivalent to a (2) collapsed lightweight inference-time network. This means, at the cost of a longer training time of the expanded network, the task metrics of the efficient inference-time network are improved. Recent works have adopted SRP for handcrafting efficient deep neural networks and have proposed universal multi-branch SRP blocks to expand only the convolutional (CONV) layers during training. Furthermore, the training schemes involve uniformly expanding every CONV layer of a given DNN, which becomes impractical for modern, large DNNs. This work presents a simple and effective technique to structurally reparametrize the dense layers, unlocking additional potential for the technique. We achieve superior accuracy improvements with our proposed Reparametrizable Fully Connected RepFC block, by expanding only dense layers of the classification head in SOTA DNNs as opposed to expanding all the CONV layers in related works. Additionally, we showcase how the RepFC block can be leveraged in transformers, where dense layers are employed throughout the network. Transformer architectures Swin-tiny, EfficientViT-B1 and Fast-T8 show a 0.2, 0.3 and 0.75 p.p. improvement respectively, on the ImageNet dataset. Our experiments demonstrate a 1.46, 0.85 and 1.14 p.p. improvements in AlexNet, MobileNet-V3-L and EfficientNet-B0 respectively on the ImageNet dataset, with zero overhead in inference time. Code is available at https://github.com/shamvbs/RepFC.

External IDs:dblp:conf/cvpr/SampathFTFFMVFS25