Regularizing Activation Distribution for Ultra Low-bit Quantization-Aware Training of MobileNets

Seongmin Park, Wonyong Sung, Jungwook Choi

Abstract: MobileNets are a family of CNN architectures that are designed for parameter efficient model deployment. However, quantization aware training (QAT) of MobileNets for lowprecision model development has not been very successful partly due to the parameter efficient nature of the model. The activation signal in MobileNet shows fairly large magnitude diversity for each channel, which can lead to poor quantization results. We address this issue by including a new loss term for QAT, which regularizes the diversity of the activation signal in each channel. The proposed method improves the accuracy of the state-of-theart QAT when evaluated on MobileNet with CIFAR10/100, and ImageNet datasets.