Special solutions with small volume exist

Tausifa Jan Saleem, Ramanjit Ahuja, Surendra Prasad, Brejesh Lall

Keywords: Neural network pruning, lottery ticket hypothesis, iterative magnitude pruning, loss landscape

Abstract:

The Lottery Ticket Hypothesis for deep neural networks emphasizes the importance of initialization used to re-train the sparser networks
obtained using the iterative magnitude pruning process. An explanation for why the specific initialization proposed by the lottery ticket hypothesis tends to work better in terms of generalization (and training) performance has been lacking. In this work, we attempt to provide insight into this phenomenon by empirically studying the volume/geometry and loss landscape characteristics of the solutions obtained at various stages of the iterative magnitude pruning process.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Style Files: I have used the style files.

Submission Number: 63