NAS-Bench-360: Benchmarking Diverse Tasks for Neural Architecture Search

Renbo Tu, Mikhail Khodak, Nicholas Carl Roberts, Nina Balcan, Ameet Talwalkar

Keywords: automated machine learning, neural architecture search

TL;DR: We provide a benchmark for neural architecture search on a diverse set of understudied tasks.

Abstract: Most existing neural architecture search (NAS) benchmarks and algorithms prioritize performance on well-studied tasks, focusing on computer vision datasets such as CIFAR and ImageNet. However, the applicability of NAS approaches in other areas is not adequately understood. In this paper, we present NAS-Bench-360, a benchmark suite for evaluating state-of-the-art NAS methods on less-explored datasets. To do this, we organize a diverse array of tasks, from classification of simple deformations of natural images to predicting protein folding and partial differential equation (PDE) solving. Our evaluation pipeline compares architecture search spaces of different flavors, and reveals varying performance on different tasks, providing baselines for further use. All data and reproducible evaluation code are open-source and publicly available. The results of our evaluation show that current state-of-the-art NAS methods often struggle to compete with simple baselines and human-designed architectures on the majority of tasks in our benchmark. At the same time, they can be quite effective on a few individual, understudied tasks. This demonstrates the importance of evaluation on diverse tasks to better understand the usefulness of different approaches to architecture search and automation.

URL: https://rtu715.github.io/NAS-Bench-360/

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2110.05668/code)