Learning Internal Dense But External Sparse Structures of Deep Neural Network

Yiqun Duan

Abstract: Recent years have witnessed two seemingly opposite developments of deep convolutional neural networks (CNNs). On one hand, increasing the density of CNNs by adding cross-layer connections achieve higher accuracy. On the other hand, creating sparsity structures through regularization and pruning methods enjoys lower computational costs. In this paper, we bridge these two by proposing a new network structure with locally dense yet externally sparse connections. This new structure uses dense modules, as basic building blocks and then sparsely connects these modules via a novel algorithm during the training process. Experimental results demonstrate that the locally dense yet externally sparse structure could acquire competitive performance on benchmark tasks (CIFAR10, CIFAR100, and ImageNet) while keeping the network structure slim.

Keywords: Convolutional Neural Network, Hierarchical Neural Architecture, Structural Sparsity, Evolving Algorithm

TL;DR: In this paper, we explore an internal dense yet external sparse network structure of deep neural networks and analyze its key properties.