Exploring Complicated Search Spaces with Interleaving-Free Sampling

Yunjie Tian, Lingxi Xie, Jiemin Fang, Qixiang Ye, Jianbin Jiao, Qi Tian

Keywords: Complicated search spaces, Interleaving-free sampling, Interleaved connections, Degradation

Abstract: The design of search space plays a crucial role in neural architecture search (NAS). Existing search spaces mostly involve short-distance connections arguably due to the increasing difficulty brought by long-distance ones. This paper systematically studies this problem in the context of super-network optimization, and reveals that the interleaved connections introduce significant noises to the amortized accuracy. Based on the observation, we propose a simple yet effective interleaving-free sampling algorithm to aid the search process, and name our method as IF-NAS. In each iteration, IF-NAS samples a sub-network that does not contain any interleaved connections. We design a difficult search space with a large number of complicatedly connected nodes, $10^{186}\times$ larger than the DARTS space, on which IF-NAS outperforms other competitors evidently. IF-NAS also generalizes to the known (easier) search spaces including DARTS and GOLD-NAS, the design of which carries great prior knowledge. Our research sheds light on extending the macro structure which is well acknowledged as a major challenge of NAS.

One-sentence Summary: We propose IF-NAS to explore complicated search spaces, where most search methods degrade because of the difficulties caused by interleaved connections.