Learning Morphisms with Gauss-Newton Approximation for Growing Networks

Neal Gregory Lawton, Aram Galstyan, Greg Ver Steeg

Keywords: Optimization, Neural Architecture Search, Growing Networks, Second-Order Methods, Gauss-Newton Approximation

TL;DR: We present a NAS algorithm that learns and evaluates network morphisms by optimizing a Gauss-Newton approximation of the loss.

Abstract: An appealing method for Neural Architecture Search (NAS) is based on growing networks via small local changes to the network’s architecture called network morphisms. These methods start with a small seed network and progressively grow the network by adding new neurons in an automated way. However, efficiently determining the best way to grow the network remains a challenge. Here we propose a NAS method for growing a network which uses a Gauss-Newton approximation of the loss function to efficiently learn and evaluate candidate network morphisms. We then optimize this approximate loss function to efficiently learn morphism parameters. We compare our method with similar NAS methods for CIFAR-10 and CIFAR-100 classification tasks, and conclude our method learns similar quality or better architectures at a smaller computational cost.

Submission Number: 75