GraphPNAS: Learning Probabilistic Graph Generators for Neural Architecture Search

Muchen Li, Jeffrey Yunfan Liu, Leonid Sigal, Renjie Liao

Abstract: Neural architectures can be naturally viewed as computational graphs. Motivated by this perspective, we, in this paper, study neural architecture search (NAS) through the lens of learning graph generative models. In contrast to existing NAS methods which largely focus on searching for a single best architecture, i.e, point estimation, we propose GraphPNAS a deep graph generative model that learns a distribution of well-performing architectures. Relying on graph neural networks (GNNs), our GraphPNAS can better capture topologies of good neural architectures and relations between operators therein. Moreover, our graph generator leads to a learnable probabilistic search method that is more flexible and efficient than the commonly used RNN generator and random search methods. Finally, we learn our generator via an efficient reinforcement learning formulation for NAS. To assess the effectiveness of our GraphPNAS, we conduct extensive experiments on four search spaces, including the challenging RandWire on TinyImageNet, ENAS on CIFAR10, and NAS-Bench-101/201. We show that our proposed graph generator consistently outperforms RNN-based one and achieves better or comparable performances than state-of-the-art NAS methods.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission: Following the Action Editors suggestions, we add DrNAS [1] baseline and optimal upper-bound optimal upper-bound results for NAS-BENCH-201 in Table 4. We also update discussion in section 4.3 to address this issue of nas-bench-201. We update the paper with discussed related works by the reviewers in the rebuttal phase. We further revise some expression and fix some error in the paper. We open sourced the code for reproducibility.

Code: https://github.com/DSL-Lab/GraphPNAS

Assigned Action Editor: ~Laurent_Charlin1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 1172