Selective Convolutional Units: Improving CNNs via Channel Selectivity

Jongheon Jeong, Jinwoo Shin

Abstract: Bottleneck structures with identity (e.g., residual) connection are now emerging popular paradigms for designing deep convolutional neural networks (CNN), for processing large-scale features efficiently. In this paper, we focus on the information-preserving nature of identity connection and utilize this to enable a convolutional layer to have a new functionality of channel-selectivity, i.e., re-distributing its computations to important channels. In particular, we propose Selective Convolutional Unit (SCU), a widely-applicable architectural unit that improves parameter efficiency of various modern CNNs with bottlenecks. During training, SCU gradually learns the channel-selectivity on-the-fly via the alternative usage of (a) pruning unimportant channels, and (b) rewiring the pruned parameters to important channels. The rewired parameters emphasize the target channel in a way that selectively enlarges the convolutional kernels corresponding to it. Our experimental results demonstrate that the SCU-based models without any postprocessing generally achieve both model compression and accuracy improvement compared to the baselines, consistently for all tested architectures.

Keywords: convolutional neural networks, channel-selectivity, channel re-wiring, bottleneck architectures, deep learning

TL;DR: We propose a new module that improves any ResNet-like architectures by enforcing "channel selective" behavior to convolutional layers

Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10)