Simple CNN for Vision
Primary Area: representation learning for computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: backbone; cnn
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Traditional Convolutional Neural Networks (CNNs) tend to use 3$\times$3 small kernels, but can only capture neighboring spatial information in one block.
Inspired by the success of Vision Transformers (ViTs) in capturing long-range visual dependencies, recent CNNs have reached a consensus on utilizing large kernel convolutions (e.g., 31$\times$31 and, astonishingly, 51$\times$51 kernels).
Nevertheless, these approaches necessitate adopting specialized techniques such as re-parameterization or sparsity, which require extra post-processing. And too large kernels are unfriendly to hardware.
This paper introduces a Simple Convolutional Neural Network (SCNN) that employs a sequence of stacked 3$\times$3 convolutions but surpasses state-of-the-art CNNs utilizing larger kernels. Notably, we propose simple yet highly effective designs that enable 3$\times$3 convolutions to progressively capture visual cues of various sizes, thereby overcoming the limitations of smaller kernels.
First, we build a thin and deep model, which encourages more convolutions to capture more spatial information under the same computing complexity instead of opting for a heavier, shallower architecture. Furthermore, we introduce an innovative block comprising two 3$\times$3 depthwise convolutions to enlarge the receptive field. Finally, we replace the input of the popular Sigmoid Linear Unit (SiLU) activation function with global average pooled features to capture all spatial information.
Our SCNN performs superior to state-of-the-art CNNs and ViTs across various tasks, including ImageNet-1K image classification, COCO instance segmentation, and ADE20K semantic segmentation.
Remarkably, SCNN outperforms the small version of Swin Transformer, a well-known ViTs, while requiring only 50\% computation, which further proves that large kernel convolution is not the only choice for high-performance CNNs.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 7100
Loading