Single-Domain Generalized Predictor for Neural Architecture Search System

Lianbo Ma, Haidong Kang, Guo Yu, Qing Li, Qiang He

Abstract:

Performance predictors are used to reduce architecture evaluation costs in neural architecture search, which
however suffers from a large amount of budget consumption
in annotating substantial architectures trained from scratch.
Hence, how to leverage existing annotated architectures to train a
generalized predictor to find the optimal architecture on unseen
target search spaces becomes a new research topic. To solve
this issue, we propose a Single-Domain Generalized Predictor
(SDGP), which aims to make the predictor only trained on a single
source search space but perform well on target search spaces. In
meta-learning, we firstly adopt feature extractor in learning the
domain-invariant features of the architectures. Then, a neural
predictor is trained to map the architectures to the accuracy of
the candidate architectures over the target domain simulated on
the source search space. Moreover, a novel multi-head attention
driven regularizer is designed to regulate the predictor to further
improve the generalization ability of the predictor for the feature
extractor. A series of experimental results have shown that the
proposed predictor outperforms the state-of-the-art predictors
in generalization and achieves significant performance gains in
finding the optimal architectures with test error 2.40% on CIFAR-
10 and 23.20% on ImageNet1k within 0.01 GPU days.