- TL;DR: A NAS benchmark applicable to almost any NAS algorithms.
- Abstract: Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years. It could be time to take a step back and analyze the good and bad aspects in the field of NAS. A variety of algorithms search architectures under different search space. These searched architectures are trained using different setups, e.g., hyper-parameters, data augmentation, regularization. This raises a fairness problem when comparing the performance of various NAS algorithms. In this work, we propose an Algorithm-Agnostic NAS Benchmark (AA-NAS-Bench) with a fixed search space, which provides a unified benchmark for almost any up-to-date NAS algorithms. The design of our search space is inspired from that used in the most popular cell-based searching algorithms, where a cell is represented as a directed acyclic graph. Each edge here is associated with an operation selected from a predefined operation set. For it to be applicable for all NAS algorithms, the search space defined in AA-NAS-Bench includes 4 nodes and 5 associated operation options, which generates 15,625 neural cell candidates in total. The training log using the same setup and performance for each architecture candidate are provided for three datasets. This allows researchers to avoid unnecessary repetitive training for selected architecture and focus solely on the search algorithm itself. The training time saved for every architecture also largely improves the efficiency of most NAS algorithms and presents a more computational cost friendly NAS community for a broader range of researchers. Side information such as fine-grained loss and accuracy is also provided, which can give inspirations to new designs of NAS algorithms. We demonstrate the applicability of the proposed AA-NAS-Bench via benchmarking many recent NAS algorithms.
- Keywords: Neural Architecture Search, AutoML, Benchmark
- Code: https://anonymous.4open.science/repository/9aa95a13-7e6a-48ed-9c77-4ac9111f7ae9/README.md