Searching for Stage-wise Neural Graphs In the Limit

Xin Zhou, Dejing Dou, Boyang Li

Keywords: neural architecture search, graphon, random graphs

TL;DR: Graphon is a good search space for neural architecture search and empirically produces good networks.

Abstract: Search space is a key consideration for neural architecture search. Recently, Xie et al. (2019a) found that randomly generated networks from the same distribution perform similarly, which suggest we should search for random graph distributions instead of graphs. We propose graphon as a new search space. A graphon is the limit of Cauchy sequence of graphs and a scale-free probabilistic distribution, from which graphs of different number of vertices can be drawn. This property enables us to perform NAS using fast, low-capacity models and scale the found models up when necessary. We develop an algorithm for NAS in the space of graphons and empirically demonstrate that it can find stage-wise graphs that outperform DenseNet and other baselines on ImageNet.