KernelWarehouse: Rethinking the Design of Dynamic Convolution

Chao Li, Anbang Yao

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: dynamic convolution, convolutional neural network, image classification, object detection

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: This paper proposes KernelWarehouse, a more general form of dynamic convolution, which strikes a favorable trade-off between parameter efficiency and representation power.

Abstract: Dynamic convolution learns a linear mixture of $n$ static kernels weighted with their sample-dependent attentions, demonstrating superior performance than normal convolution. However, it increases the number of convolutional parameters by $n$ times. This and the optimization difficulty lead to no research progress that can allow researchers to use a significant large value of $n$ (e.g., $n>100$ instead of typical setting $n<10$) to push forward the performance boundary of dynamic convolution while enjoying parameter efficiency. To fill this gap, in this paper, we rethink the basic concepts of **"kernels"**, **"assembling kernels"** and **"attention function"** in dynamic convolution through the lens of exploiting convolutional parameter dependencies within the same layer and across successive layers, and propose **"KernelWarehouse"**. As a more general form of dynamic convolution, KernelWarehouse provides a high degree of freedom to fit a desired parameter budget under large kernel numbers (e.g., $n=108$ or $n=188$). We validate our method on ImageNet and MS-COCO datasets with various ConvNet architectures, and show that it attains state-of-the-art results. For instance, the ResNet18$|$ResNet50$|$MobileNetV2$|$ConvNeXt-Tiny model trained with KernelWarehouse on ImageNet reaches 76.05\%$|$81.05\%$|$75.92\%$|$82.55\% top-1 accuracy. Thanks to its flexible design, KernelWarehouse can even reduce the model size of a ConvNet while improving the accuracy, e.g., our ResNet18 model with 36.45\%$|$65.10\% parameter reduction to the baseline shows 2.89\%$|$2.29\% absolute improvement to top-1 accuracy. Code is provided for results reproduction.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1252