Search Spaces for Neural Model Training

Darko Stosic, Dusan Stosic

Keywords: search space, sparsity, neural models, deep learning

Abstract: While larger neural models are pushing the boundaries of what deep learning can do, often more weights are needed to train models rather than to run inference for tasks. This paper seeks to understand this behavior using search spaces -- adding weights creates extra degrees of freedom that form new paths for optimization (or wider search spaces) rendering neural model training more effective. We then show how we can augment search spaces to train sparse models attaining competitive scores across dozens of deep learning workloads. They are also are tolerant of structures targeting current hardware, opening avenues for training and inference acceleration. Our work encourages research to explore beyond massive neural models being used today.

One-sentence Summary: This paper seeks to understand why using more weights improves neural model training in the context of search spaces and using these insights to improve sparse training.

Community Implementations: [ 4 code implementations](https://www.catalyzex.com/paper/arxiv:2105.12920/code)