Meta Architecture Search

Albert E Shaw, Wei Wei, Weiyang Liu, Le Song, Bo Dai

Abstract: Neural Architecture Search (NAS) has been quite successful in constructing state-of-the-art models on a variety of tasks. Unfortunately, the amount of computational cost makes it difficult to scale to a number of tasks. In this paper, we make the first attempt to study Meta Architecture Search, which aims at learning a task-agnostic representation that will be used to speed up the process of architecture search on a large number of tasks. We propose Bayesian Meta Architecture SEarch~(BASE), which takes advantage of Bayesian formulation of the architecture search problem that learns over an entire set of tasks simultaneously and quickly find optimal architectures for even previously unseen tasks. We show that on Imagenet our model is able to achieve 25.71% and 8.08% on top-1 error and top-5 error respectively using an adaptation time of under an hour on an 8 GPU days trained meta network. This result beats the state-of-the-art methods such as DARTS and SNAS significantly in terms of both performance and computational costs. Our model can also find competitive models with just single-shot adaptation on unseen datasets. Our research opened new possibilities on efficient and massively salable architecture search research across multiple tasks.

Code Link: https://github.com/ashaw596/meta_architecture_search

CMT Num: 6001