Feature Map Similarity Reduction in Convolutional Neural Networks

TMLR Paper3595 Authors

Abstract: It has been observed that Convolutional Neural Networks (CNNs) suffer from redundancy in feature maps, leading to inefficient capacity utilization. Efforts to address this issue have largely focused on kernel orthogonality method. In this work, we theoretically and empirically demonstrate that kernel orthogonality does not necessarily lead to a reduction in feature map redundancy. Based on this analysis, we propose the Convolutional Similarity method to reduce feature map similarity, independently of the CNN's input. The Convolutional Similarity can be minimized as either a regularization term or an iterative initialization method. Experimental results show that minimizing Convolutional Similarity not only improves classification accuracy but also accelerates convergence. Furthermore, our method enables the use of significantly smaller models to achieve the same level of performance, promoting a more efficient use of model capacity. Future work will focus on coupling the iterative initialization method with the optimization momentum term and examining the method's impact on generative frameworks.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

- Fixed a table column
- Added a statement about the derivation of the theoretical results using Parseval's and the convolution theorems

Assigned Action Editor: ~Zhiding_Yu1

Submission Number: 3595