Hyperflux: Pruning Reveals the Importance of Weights

Eugen Barbulescu, Antonio Alexoaie, Lucian Busoniu

Keywords: Computer Vision, Architectures, Algorithms, Sparsity

TL;DR: We propose an empirically strong dynamic pruning method which focuses on conceptual grounding.

Abstract: Network pruning is used to reduce inference latency and power consumption in large neural networks. However, most existing methods use ad-hoc heuristics, lacking much insight and justified mainly by empirical results. We introduce Hyperflux, a conceptually-grounded $L_0$ pruning approach that estimates each weight’s importance through its *flux*, the gradient's response to the weight's removal. A global *pressure* term continuously drives all weights toward pruning, with those critical for accuracy being automatically regrown based on their flux. We postulate several properties that naturally follow from our framework and experimentally validate each of them. One such property is the relationship between final sparsity and pressure, for which we derive a generalized scaling-law equation that is used to design our sparsity-controlling scheduler. Empirically, we demonstrate state-of-the-art results with ResNet-50 and VGG-19 on CIFAR-10 and CIFAR-100.

Primary Area: Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Submission Number: 17073