GradMax: Growing Neural Networks using Gradient InformationDownload PDF

Sep 29, 2021 (edited May 06, 2022)ICLR 2022 PosterReaders: Everyone
  • Keywords: efficient training, efficient, computer vision, architecture search
  • Abstract: The architecture and the parameters of neural networks are often optimized independently, which requires costly retraining of the parameters whenever the architecture is modified. In this work we instead focus on growing the architecture without requiring costly retraining. We present a method that adds new neurons during training without impacting what is already learned, while improving the training dynamics. We achieve the latter by maximizing the gradients of the new weights and efficiently find the optimal initialization by means of the singular value decomposition (SVD). We call this technique Gradient Maximizing Growth (GradMax) and demonstrate its effectiveness in variety of vision tasks and architectures. We open sourced our code at https://github.com/google-research/growneuron
  • One-sentence Summary: We present a method that adds new neurons during training without impacting what is already learned, while improving the training dynamics.
14 Replies

Loading