Tiered Pruning for Efficient Differentialble Inference-Aware Neural Architecture Search

Slawomir Kierat, Mateusz Sieniawski, Denys Fridman, Chenhan D. Yu, Szymon Migacz, Pawel Morkisz, Alex Fit-Florea

Keywords: nas, dnas, neural architecture search, differentiable neural architecture search, state-of-the-art, imagenet, classification, gpunet, efficientnet, pruning, inference-aware, computer vision, object detection

Abstract: We propose three novel pruning techniques to improve the cost and results of inference-aware Differentiable Neural Architecture Search (DNAS). First, we introduce $\textbf{Prunode}$, a stochastic bi-path building block for DNAS, which can search over inner hidden dimensions with $\mathcal{O}(1)$ memory and compute complexity. Second, we present an algorithm for pruning blocks within a stochastic layer of the SuperNet during the search. Third, we describe a novel technique for pruning unnecessary stochastic layers during the search. The optimized models resulting from the search are called PruNet and establishes a new state-of-the-art Pareto frontier for NVIDIA V100 in terms of inference latency for ImageNet Top-1 image classification accuracy. PruNet as a backbone also outperforms GPUNet and EfficientNet on the COCO object detection task on inference latency relative to mean Average Precision (mAP).

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/tiered-pruning-for-efficient-differentialble/code)