NAS-Bench-Zero: A Large Scale Dataset for Understanding Zero-Shot Neural Architecture Search

Hanlin Chen, Ming Lin, Xiuyu Sun, Hao Li

Abstract: Zero-shot Neural Architecture Search (ZS-NAS) is a recently developed low-cost NAS framework which identifies top-performer neural architectures from a large candidate pool without training their parameters. Despite its popularity in recent NAS literatures, the effectiveness of ZS-NAS has not been comprehensively understood. Previous works analyze ZS-NAS methods on NAS benchmark datasets such as NAS-Bench-101/201/301 which are initially designed for learning network topology with irregular connections. However, most modern state-of-the-art networks as well as popular classical ones are designed in more conventional, well-established search spaces such as ResNet (RS) and MobileNet (MB) search space. This imposes a significant gap between the benchmark dataset and real-world practice, hindering a deeper understanding of ZS-NAS. In this work, we aim to bridge the gap systematically. First, we collect a novel large-scale dataset termed NAS-Bench-Zero for benchmarking and understanding popular ZS-NAS methods in the conventional RS/MB search spaces. Then the characteristics of these ZS-NAS methods are extensively examined from various aspects. Notably, we find that: 1) the performance of ZS-NAS on NAS-Bench-101/201/301 cannot transfer to RS/MB search spaces; 2) A proxy with higher ranking correlation score may actually perform worse in constrained NAS; 3) existing zero-shot proxies cannot outperform naive proxies such as FLOPs/params in RS/MB search spaces; 4) Top best zero-shot proxies as well as FLOPs/params compensate each other. Based on these new discoveries, we propose i) a novel hybrid zero-shot proxy which outperforms existing ones by a large margin and is transferable among popular search spaces; ii) a new index for better measuring the true performance of ZS-NAS proxies in constrained NAS. Source code and the NAS-Bench-Zero dataset will be released after publication.