Automated Super-Network Generation for Scalable Neural Architecture SearchDownload PDF

25 Feb 2022, 12:35 (modified: 16 Jul 2022, 13:35)AutoML-Conf 2022 (Main Track)Readers: Everyone
Abstract: Weight-sharing Neural Architecture Search (NAS) solutions often discover neural network architectures that outperform their human-crafted counterparts. Weight-sharing allows the creation and training of super-networks that contain many smaller and more efficient child models, a.k.a., sub-networks. For an average deep learning practitioner, generating and training one of these super-networks for an arbitrary neural network architecture design space can be a daunting experience. In this paper, we present BootstrapNAS, a software framework that addresses this challenge by automating the generation and training of super-networks. Developers can use this solution to convert a pre-trained model into a super-network. BootstrapNAS then trains the super-network using a weight-sharing NAS technique available in the framework or provided by the user. Finally, a search component discovers high-performing sub-networks that are returned to the end-user. We demonstrate BootstrapNAS by automatically generating super-networks from popular pre-trained models (MobileNetV2, MobileNetV3, EfficientNet, ResNet50 and HyperSeg), available from Torchvision and other repositories. BootstrapNAS can achieve up to 9.87× improvement in throughput in comparison to the pre-trained Torchvision ResNet-50 (FP32) on Intel Xeon platform.
Keywords: Neural Architecture Search, AutoML, Model Compression, Super-network
One-sentence Summary: The paper describes a software framework that automates the generation and training of super-networks from pre-trained models, and the subsequent search for high-performing sub-networks, enabling scalable Neural Architecture Search.
Track: Main track
Reproducibility Checklist: Yes
Broader Impact Statement: Yes
Paper Availability And License: Yes
Code Of Conduct: Yes
Reviewers: Pablo Munoz,
Main Paper And Supplementary Material: pdf
Code And Dataset Supplement: zip
CPU Hours: 1800
GPU Hours: 2000
TPU Hours: 0
Evaluation Metrics: Yes
Estimated CO2e Footprint: 167
Class Of Approaches: Evolutionary Methods
Datasets And Benchmarks: ImageNet, CIFAR10, CIFAR100, VFX
Performance Metrics: Accuracy, latency, MACs
6 Replies