Prior Knowledge Guided Neural Architecture Generation

Jingrong Xie, Han Ji, Yanan Sun

Abstract: Automated architecture design methods, especially neural architecture search, have attracted increasing attention. However, these methods naturally need to evaluate numerous candidate architectures during the search process, thus computationally extensive and time-consuming. In this paper, we propose a prior knowledge guided neural architecture generation method to generate high-performance architectures without any search and evaluation process. Specifically, in order to identify valuable prior knowledge for architecture generation, we first quantify the contribution of each component within an architecture to its overall performance. Subsequently, a diffusion model guided by prior knowledge is presented, which can easily generate high-performance architectures for different computation tasks. Extensive experiments on new search spaces demonstrate that our method achieves superior accuracy over state-of-the-art methods. For example, we only need $0.004$ GPU Days to generate architecture with $76.1\%$ top-1 accuracy on ImageNet and $97.56\%$ on CIFAR-10. Furthermore, we can find competitive architecture for more unseen search spaces, such as TransNAS-Bench-101 and NATS-Bench, which demonstrates the broad applicability of the proposed method.

Lay Summary: We try to design architectures automatically for different tasks. However, it is hard to train a model that can generate architectures automatically with few computation costs. In this paper, we learn some high-performance architectures to obtain prior knowdge. And with this knowledge, we can train a model that can generate optimal architectures. In many datasets, our methods show good performance.

Primary Area: Deep Learning->Everything Else

Keywords: neural architecture search, neural architecture generation

Submission Number: 9493