Hyperflows: Pruning Reveals the Importance of Weights

Barbulescu Eugen, Antonio Alexoaie

TL;DR: We propose a dynamic pruning method, introducing notions of weight flow, pressure and derive "neural pruning laws".

Abstract: Network pruning is used to reduce inference latency and power consumption in large neural networks. However, most existing methods struggle to accurately assess the importance of individual weights due to their inherent interrelatedness, leading to poor performance, especially at extreme sparsity levels. We introduce *Hyperflows*, a dynamic pruning approach that estimates each weight’s importance by observing the network’s gradient response to the weight's removal. A global *pressure* term continuously drives all weights toward pruning, with those critical for accuracy being automatically regrown based on their *flow*, the aggregated gradient signal when they are absent. We explore the relationship between final sparsity and pressure, deriving power-law equations similar to those found in neural scaling laws. Empirically, we demonstrate state-of-the-art results with ResNet-50 and VGG-19 on CIFAR-10 and CIFAR-100.

Primary Area: Deep Learning->Algorithms

Keywords: Computer Vision, Architectures, Algorithms, Sparsity

Submission Number: 7280