- Abstract: One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice. Nevertheless, there is still a lack of understanding on how these weight-sharing algorithms exactly work due to the many factors controlling the dynamics of the process. In order to allow a scientific study of these components, we introduce a general framework for one-shot NAS that can be instantiated to many recently-introduced variants and introduce a general benchmarking framework that draws on the recent large-scale tabular benchmark NAS-Bench-101 for cheap anytime evaluations of one-shot NAS methods. To showcase the framework, we compare several state-of-the-art one-shot NAS methods, examine how sensitive they are to their hyperparameters and how they can be improved by careful regularization, and compare their per- formance to that of blackbox optimizers for NAS-Bench-101.
- Code: https://github.com/AnonymousMetaLearn/Towards-benchmarking-and-dissecting-one-shot-neural-architecture-search
- Keywords: Neural Architecture Search, Deep Learning, Computer Vision