One-Shot Neural Architecture Search via Compressive Sensing

Minsu Cho, Mohammadreza Soltani, Chinmay Hegde

Keywords: deep learning, autoML, neural architecture search, image classification, language modeling

TL;DR: A new approach for one-shot neural architecture search that blends in techniques from Fourier-sparse recovery.

Abstract: Neural architecture search (NAS), or automated design of neural network models, remains a very challenging meta-learning problem. Several recent works (called "one-shot" approaches) have focused on dramatically reducing NAS running time by leveraging proxy models that still provide architectures with competitive performance. In our work, we propose a new meta-learning algorithm that we call CoNAS, or Compressive sensing-based Neural Architecture Search. Our approach merges ideas from one-shot NAS approaches with iterative techniques for learning low-degree sparse Boolean polynomial functions. We validate our approach on several standard test datasets, discover novel architectures hitherto unreported, and achieve competitive (or better) results in both performance and search time compared to existing NAS approaches. Further, we provide theoretical analysis via upper bounds on the number of validation error measurements needed to perform reliable meta-learning; to our knowledge, these analysis tools are novel to the NAS literature and may be of independent interest.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1906.02869/code)