Balanced Mixture of Supernets for Learning the CNN Pooling Architecture

Mehraveh Javan Roshtkhari, Matthew Toews, Marco Pedersoli

Keywords: CNN, Neural architecture search, supernet, pooling layers, weight-sharing

TL;DR: We present a novel neural architecture search method using a balanced mixture of supernets to find the pooling locations for CNN

Abstract:

Downsampling layers, including pooling and strided convolutions, are crucial components of the convolutional neural network architecture that determine both the granularity/scale of image feature analysis as well as the receptive field size of a given layer. To fully understand this problem we analyse the performance of models independently trained with each pooling configurations on CIFAR10, using a ResNet20 network and show that the position of the downsampling layers can highly influence the performance of a network and predefined downsampling configurations are not optimal.
Network Architecture Search (NAS) might be used to optimize downsampling configurations as an hyperparameter. However, we find that common one-shot NAS based on a single SuperNet do not work for this problem. We argue that this is because a SuperNet trained for finding the optimal pooling configuration fully shares its parameters among all pooling configurations. This makes its training hard because learning some configurations can harm the performance of others.
Therefore, we propose a balanced mixture of SuperNets that automatically associates pooling configurations to different weight models and helps to reduce the weight-sharing and interinfluence of pooling configurations on the SuperNet parameters. We evaluate our proposed approach on CIFAR10, CIFAR100, as well as Food101, and show that in all cases our model outperforms other approaches and improves over the default pooling configurations.

Submission Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Reviewers: Yes

CPU Hours: 0

GPU Hours: 1840

TPU Hours: 0

Evaluation Metrics: No

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/balanced-mixture-of-supernets-for-learning/code)