APP: Anytime Progressive PruningDownload PDF

Published: 18 Nov 2022, Last Modified: 22 Oct 2023CLL@ACML2022Readers: Everyone
Keywords: Network Pruning, Dynamic Architectures, Anytime Learning
TL;DR: We propose a novel way of progressive pruning, termed as Anytime Progressive Pruning (APP) to answer the following question: “Given a dense neural network and a target sparsity, what should be the optimal way of pruning the model in ALMA setting?”
Abstract: With the latest advances in deep learning, several methods have been investigated for optimal learning settings in scenarios where the data stream is continuous over time. However, training sparse networks in such settings has often been overlooked. In this paper, we explore the problem of training a neural network with a target sparsity in a particular case of online learning: the anytime learning at macroscale paradigm (ALMA). We propose a novel way of progressive pruning, referred to as \textit{Anytime Progressive Pruning} (APP); the proposed approach significantly outperforms the baseline dense and Anytime OSP models across multiple architectures and datasets under short, moderate, and long-sequence training. Our method, for example, shows an improvement in accuracy of $\approx 7\%$ and a reduction in the generalization gap by $\approx 22\%$, while being $\approx 1/3$ rd the size of the dense baseline model in few-shot restricted imagenet training.
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 1 code implementation](https://www.catalyzex.com/paper/arxiv:2204.01640/code)
1 Reply

Loading