Intra-layer Neural Architecture Search

Dong Kai Wang, Nam Sung Kim

Keywords: Neural Networks, Neural Architecture Search, Training Algorithms

Abstract: We propose an efficient neural architecture search (NAS) algorithm with a flexible search space that encompasses layer operations down to individual weights. This work addresses NAS challenges in a search space of weight connections within layers, specifically the large number of architecture variations compared to a high-level search space with predetermined layer types. Our algorithm continuously evolves network architecture by adding new candidate parameters (weights and biases) using a first-order estimation based on their gradients at 0. Training is decoupled into alternating steps: adjusting network weights holding architecture constant, and adjusting network architecture holding weights constant. We explore additional applications by extend this method for multi-task learning with shared parameters. On the CIFAR-10 dataset, our evolved network achieves an accuracy of 97.42\% with 5M parameters, and 93.75\% with 500K parameters. On the ImageNet dataset, we achieve 76.6\% top-1 and 92.5\% top-5 accuracy with a search restriction of 8.5M parameters.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: Neural architecture search at the level of individual weight parameters.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=uDA-FjuuB3