Rethinking Compressed Convolution Neural Network from a Statistical Perspective

Feiqing Huang, Yuefeng Si, Guodong Li

Keywords: Compressed Convolutional Neural Network, Tensor Decomposition, Sample Complexity Analysis

Abstract: Many designs have recently been proposed to improve the model efficiency of convolutional neural networks (CNNs) at a fixed resource budget, while there is a lack of theoretical analysis to justify them. This paper first formulates CNNs with high-order inputs into statistical models, which have a special "Tucker-like" formulation. This makes it possible to further conduct the sample complexity analysis to CNNs as well as compressed CNNs via tensor decomposition. Tucker and CP decompositions are commonly adopted to compress CNNs in the literature. The low rank assumption is usually imposed on the output channels, which according to our study, may not be beneficial to obtain a computationally efficient model while a similar accuracy can be maintained. Our finding is further supported by ablation studies on CIFAR10, SVNH and UCF101 datasets.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We theoretically explore the mechanism of tensor factorized convolutional neural networks.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=c3HD190Z7Y