Structured Lottery Ticket Hypothesis for Effective Deep Neural Network Model Size Reduction

Arijit Ukil, Ishan Sahu, Mridul Biswas, Arpan Pal, Angshul Majumdar

Abstract: Lottery ticket hypothesis (LTH) has supporting evidence that a subset of network parameters is able to provide similar performance of large deep neural network models. While pruning is implicit in LTH, the explicit benefit of model size reduction is yet to be achieved given that LTH only produces zero-weight parameters. The arbitrary removal of zero weight nodes from the model graph is non-trivial. In this paper, we propose a method of LTH-integrated model size removal by systemically eliminating the most sparsified layers after each episode of LTH iteration through structural pruning transformation. Our proposed LTH-Reduced size model imbibes the capability of effective discovery of the sparsified nodes and systemically generates smaller size models close to the performance of the base model. The smaller size model reduces the computational burden, energy consumption, and latency along with smaller memory requirements and thus, it leads us to the opportunity to develop niche and important applications in different domains including automobiles, robotics, healthcare etc.