NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks

Renbo Tu, Nicholas Roberts, Mikhail Khodak, Junhong Shen, Frederic Sala, Ameet Talwalkar

Keywords: automated machine learning, neural architecture search, diverse tasks

Abstract: Most existing neural architecture search (NAS) benchmarks and algorithms prioritize well-studied tasks, e.g. image classification on CIFAR or ImageNet. This makes the performance of NAS approaches in more diverse areas poorly understood. In this paper, we present NAS-Bench-360, a benchmark suite to evaluate methods on domains beyond those traditionally studied in architecture search, and use it to address the following question: do state-of-the-art NAS methods perform well on diverse tasks? To construct the benchmark, we curate ten tasks spanning a diverse array of application domains, dataset sizes, problem dimensionalities, and learning objectives. Each task is carefully chosen to interoperate with modern CNN-based search methods while possibly being far-afield from its original development domain. To speed up and reduce the cost of NAS research, for two of the tasks we release the precomputed performance of 15,625 architectures comprising a standard CNN search space. Experimentally, we show the need for more robust NAS evaluation of the kind NAS-Bench-360 enables by showing that several modern NAS procedures perform inconsistently across the ten tasks, with many catastrophically poor results. We also demonstrate how NAS-Bench-360 and its associated precomputed results will enable future scientific discoveries by testing whether several recent hypotheses promoted in the NAS literature hold on diverse tasks. NAS-Bench-360 is hosted at https://nb360.ml.cmu.edu.

Author Statement: Yes

TL;DR: We provide a benchmark for neural architecture search on a diverse set of understudied tasks.

URL: https://nb360.ml.cmu.edu/

Dataset Url: https://nb360.ml.cmu.edu/

License:

Code license: MIT.
Dataset licenses:
• CIFAR-100: CC BY 4.0 (on https://www.tensorflow.org/datasets/catalog/cifar100)
• Spherical CIFAR-100: CC BY-SA
• NinaPro: CC BY-ND
• FSD50k: CC BY 4.0
• Darcy Flow: MIT
• DeepCov, PSICOV: GPL
• Cosmic: CC BY 4.0
• ECG: ODC-BY 1.0
• Satellite: GPL 3.0
• Deepsea: CC BY 4.0

Contribution Process Agreement: Yes

In Person Attendance: Yes

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2110.05668/code)