Pick-or-Mix: Dynamic Channel Sampling for ConvNets
Keywords: Channel Squeezing, Network Downscaling, Dynamic Channel Sampling
TL;DR: A single dynamic channel sampling module targeting various tasks at once i.e. channel squeezing, network downscaling, dynamic pruning
Abstract: Channel squeezing is a crucial operation in convolutional neural networks (ConvNets). It is carried out via 1 × 1 convolution layers and dominates a large portion of computations and parameters of a given network. ResNet-50, for instance, consists of 16 such layers, forming 33% of total layers and 25% (1.05B/4.12B) of total FLOPs. In light of their predominance, we present a new multi-purpose module for dynamic channel sampling, namely Pick-or-Mix (PiX). PiX divides a set of channels into subsets and then picks from them, where the picking decision is dynamically made per each pixel based on the input activations. We show that PiX allows ConvNets to learn better data representation than vanilla channel squeezing in far fewer computations. We plug PiX into prominent ConvNet architectures and verify its multi-purpose utilities. After replacing 1 × 1 channel squeezing layers in the ResNet family with PiX, the networks become 25% faster without losing accuracy. We also show that PiX can achieve state-of-the-art performance on network downscaling and dynamic channel pruning.
Primary Area: representation learning for computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3223
Loading