When, where, and how to add new neurons to ANNsDownload PDF

25 Feb 2022, 12:35 (modified: 16 Jul 2022, 13:35)AutoML-Conf 2022 (Main Track)Readers: Everyone
Abstract: Neurogenesis in ANNs is an understudied and difficult problem, even compared to other forms of structural learning like pruning. By decomposing it into triggers and initializations, we introduce a framework for studying the various facets of neurogenesis: when, where, and how to add neurons during the learning process. We present the Neural Orthogonality (NORTH*) suite of neurogenesis strategies, combining layer-wise triggers and initializations based on the orthogonality of activations or weights to dynamically grow performant networks that converge to an efficient size. We evaluate our contributions against other recent neurogenesis works across a variety of supervised learning tasks.
Keywords: Machine learning, neural networks, neural architecture search
One-sentence Summary: The NORTH* suite of neurogenesis strategies determine when, where, and how to add neurons to neural networks during training.
Track: Main track
Reproducibility Checklist: Yes
Broader Impact Statement: Yes
Paper Availability And License: Yes
Code Of Conduct: Yes
Reviewers: Kaitlin Maile, kaitlin.maile@irt.fr
Main Paper And Supplementary Material: pdf
CPU Hours: 11766.703
GPU Hours: 4792.967
TPU Hours: 0
Evaluation Metrics: No
Estimated CO2e Footprint: 159.6
Class Of Approaches: Neurogenesis, Structural learning, Neural Architecture Search
Datasets And Benchmarks: MNIST, CIFAR-10, CIFAR-100
Performance Metrics: Accuracy, Network Size
8 Replies