Revisiting Loss Modelling for Unstructured Pruning

César Laurent, Camille Ballas, Thomas George, Pascal Vincent, Nicolas Ballas

Keywords: Deep Learning, Network Pruning, Unstructured Pruning

Abstract: By removing parameters from deep neural networks, unstructured pruning methods aim at cutting down memory footprint and computational cost, while maintaining prediction accuracy. In order to tackle this otherwise intractable problem, many of these methods model the loss landscape using first or second order Taylor expansions to identify which parameters can be discarded. We revisit loss modelling for unstructured pruning: we show the importance of ensuring locality of the pruning steps, and systematically compare first and second order Taylor expansions. Finally, we show that better preserving the original network function does not necessarily transfer to better performing networks after fine-tuning, suggesting that only considering the impact of pruning on the loss might not be a sufficient objective to design good pruning criteria.

One-sentence Summary: Studies the assumptions behind the loss models used in unstructured pruning.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2006.12279/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=RTfDKJsPC7