Lightweight CNNs Under A Unifying Tensor View

Jason Chun Lok Li, Rui Lin, Jiajun Zhou, Edmund Y. Lam, Ngai Wong

Keywords: compression, tensor decomposition, CNNs, FPGA

TL;DR: A unifying tensor view is introduced, which provides an easy-to-understand graphical illustration of various lightweight CNN components. A novel shift layer pruning scheme is proposed in response to the framework.

Abstract: Despite the decomposition of convolutional kernels for lightweight CNNs being well studied, previous works that relied on tensor network diagrams or higher dimensional abstraction lacked geometry intuition. Our work captures the CNN kernel as a 3D tensor and explores its various decompositions, allowing for a straightforward graphical and analytical perspective between different tensor approximation schemes and efficient CNN components, including pointwise and depthwise convolutions. Extensive experiments are conducted, showing that a pointwise-depthwise-pointwise (PDP) configuration via a canonical polyadic decomposition (CPD) initialization can be a viable starting point for lightweight CNNs. The compression ratio of VGG-16 can reach over $50\%$ while its performance outperforms its randomly initialized counterpart by $>10\%$ in terms of accuracy. FPGA experiments for the PDP model further demonstrate its hardware efficacy, namely, $2.4\times$ faster and $1.4\times$ more energy efficient than the standard conv2d. Furthermore, our framework offers a unique slice-wise illustration and is the first to ever draw a connection to the shift layer. Such insight inspires a first-of-its-kind pruning method for shift layers, achieving nearly $50\%$ compression with $<1\%$ drop in accuracy for ShiftResNet-20.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Applications (eg, speech processing, computer vision, NLP)