ShiftAddAug: Augment Multiplication-Free Tiny Neural Network with Hybrid Computation

Yipin Guo, Zihao Li, Qinyuan Ren

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: tiny neural network, multiplication-free network

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We train the multiplication-free tiny NN as a sub-model of a large multiplicative model, and use only multiplication-free part (e.g. bitwise shift and additions) for inference, resulting in higher accuracy and lower energy consumption.

Abstract:

We introduce a novel training methodology termed ShiftAddAug aimed at enhancing the performance of multiplication-free tiny neural networks. Multiplication-free operators, such as Shift and Add, have garnered attention because of their hardware-friendly nature. They are more suitable for deployment on resource-limited platforms with reduced energy consumption and computational demands.
However, multiplication-free networks usually suffer from under-performance in terms of accuracy compared to their vanilla counterpart with the same structure.
ShiftAddAug uses costly multiplication to augment efficient but less powerful multiplication-free operators, improving network accuracy without any inference overhead. It puts a multiplication-free tiny NN into a large multiplicative model and encourages it to be trained as a sub-model to obtain additional supervision, rather than as an independent model. In the process of inference, only the multiplication-free tiny model is used.
The effectiveness of ShiftAddAug is demonstrated through experiments in image classification, consistently resulting in significant improvements in accuracy and energy saving. Notably, it achieves up to a 4.05\% accuracy improvement on the CIFAR100 while simultaneously having a 68.9\% reduction in energy consumption compared to its costly multiplication counterpart.
Additionally, neural architecture search is used to obtain better augmentation effects and smaller but stronger multiplication-free tiny neural networks.
Codes and models will be released upon acceptance.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2032