NAS-Bench-Compre: A Comprehensive Neural Architecture Search Benchmark with Customizable Components

Di Wang, Kun Jing, Jungang Xu

Abstract: Neural Architecture Search (NAS) aims to discover the optimal architectures for various tasks, which developed rapidly but disorderly in the past few years. Therefore, the proposal of NAS benchmark is necessary and significant for the development of NAS. NAS has three components (search space, search strategy, and estimation strategy) to be evaluated. However, the existing benchmarks mainly evaluate search strategy only, lacking consideration of the other two components. Thus, it is difficult for current benchmarks to comprehensively evaluate the three components of NAS. In this work, we propose NAS-Bench-Compre, a comprehensive benchmark for NAS that allows users to customize search spaces, search strategies, and evaluation strategies for evaluation. Additionally, we provide essential user interfaces and implementations for each decoupled and customized component to ensure fairness and usability of NAS evaluation. We support the evaluation of the three components of the prominent NAS algorithms. Currently, we implement evaluations for DARTS, GMAENAS, OFA, AutoFormer, and Zero-shot NAS. All codes are publicly available at https://github.com/kunjing96/NAS-Bench-Compre.