Encodings for Prediction-based Neural Architecture Search

Yash Akhauri, Mohamed S Abdelfattah

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Architecture Encodings, Neural architecture search (NAS), Predictor-based NAS, AutoML

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A study on neural architecture encodings with a novel SoTA predictor, conducted on 1.5 million neural networks over 13 design spaces.

Abstract:

Predictor-based methods have substantially enhanced Neural Architecture Search (NAS) optimization, with the efficacy of these predictors largely influenced by the method of encoding neural network architectures.
While traditional encodings used an adjacency matrix describing the graph structure of a neural network, novel encodings embrace a variety of approaches from unsupervised pretraining of latent representations to vectors of zero-cost proxies.
In this paper, we categorize and investigate neural encodings from three main types: structural, learned, and score-based.
Furthermore, we extend these encodings and introduce unified encodings, that extend NAS predictors to multiple search spaces.
Our analysis draws from experiments conducted on over 1.5 million neural network architectures on NAS spaces such as NASBench-101 (NB101), NB201, NB301, Network Design Spaces (NDS), and TransNASBench-101.
Building on our study, we present our predictor FLAN: Flow Attention for NAS.
FLAN integrates critical insights on predictor design, transfer learning, and unified encodings to enable more than an order of magnitude cost reduction for training NAS accuracy predictors. Our implementation and encodings for all neural networks are open-sourced at \url{https://anonymous.4open.science/r/flan_nas-433F/}.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6329