Integrating Large Circular Kernels into CNNs through Neural Architecture Search

Kun He, Chao Li, Yixiao Yang, Gao Huang, John E. Hopcroft

Keywords: circular kernel, Convolutional Neural Network, Neural Architecture Search, operation space, large kernel

Abstract:

The square kernel is a standard unit for contemporary Convolutional Neural Networks (CNNs), as it fits well on the tensor computation for the convolution operation. However, the retinal ganglion cells in the biological visual system have approximately concentric receptive fields. Motivated by this observation, we propose using the circular kernel with a concentric and isotropic receptive field as an option for convolution operation. We first substitute the $3 \times 3$ square kernels with the corresponding circular kernels or our proposed integrated kernels in the typical ResNet architecture, and the modified models after training yield similar or even competitive performance. We then show the advantages of large circular kernels over the corresponding square kernels in that the difference and the improvement are more distinct. Hence, we speculate that large circular kernels would help find advanced neural network models by the Neural Architecture Search (NAS). To validate our hypothesis, we expand the operation space in several typical NAS methods with convolutions of large circular kernels.
Experimental results show that the searched new neural network models contain large circular kernels and significantly outperform the original searched models. The additional empirical analysis also reveals that the large circular kernel help the model to be more robust to rotated or sheared images due to its rotation invariance.

One-sentence Summary: We propose to use circular kernels for convolution in neural networks, expand the operation space of Neural Architecture Search with large circular kernels, and demonstrate its effectiveness for neural architecture search.