Better Together: Resnet-50 accuracy with $13 \times $ fewer parameters and at $3 \times $ speed

Utkarsh Nath, Shrinu Kushagra

Keywords: Deep neural networks, Theory of deep networks, deep regularization, Neural network compression

Abstract: Recent research on compressing deep neural networks has focused on reducing the number of parameters. Smaller networks are easier to export and deploy on edge-devices. We introduce Adjoined networks as a training approach that can regularize and compress any CNN-based neural architecture. Our one-shot learning paradigm trains both the original and the smaller networks together. The parameters of the smaller network are shared across both the architectures. We prove strong theoretical guarantees on the regularization behavior of the adjoint training paradigm. We complement our theoretical analysis by an extensive empirical evaluation of both the compression and regularization behavior of adjoint networks. For resnet-50 trained adjointly on Imagenet, we are able to achieve a $13.7 \times$ reduction in the number of parameters and a $3 \times$ improvement in inference time without any significant drop in accuracy. For the same architecture on CIFAR-100, we are able to achieve a $99.7 \times$ reduction in the number of parameters and a $5 \times$ improvement in inference time. On both these datasets, the original network trained in the adjoint fashion gains about $3\%$ in top-1 accuracy as compared to the same network trained in the standard fashion.

One-sentence Summary: A new paradigm to train neural network with theoretical guarentees and empirical evaluation.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=TEoMTpCUGs