Signal Collapse in One-Shot Pruning: When Sparse Models Fail to Distinguish Neural Representations

Dhananjay Saikumar, Blesson Varghese

Keywords: One-Shot pruning, Signal collapse

TL;DR: We identify a new failure more in one-shot pruning, which explains severe accuracy loss upon one-shot pruning via a new lens. We propose a novel method to rectify this and offer SOTA accuracy.

Abstract: The size of modern neural networks has made inference increasingly resource-intensive. Network pruning reduces model size by sparsifying parameters. One-shot pruning, which selects parameters via impact-based importance scores and applies second-order parameter updates, often incurs severe accuracy loss. We identify for the first time that this degradation occurs due to a phenomenon we refer to as signal collapse, which is a significant reduction in activation variance across layers, rather than the removal of `important' parameters. To address this, we introduce REFLOW, which restores layer-wise activation variance without modifying any parameters. REFLOW uncovers high-quality sparse subnetworks within the original parameter space, enabling vanilla magnitude pruning to match or exceed complex baselines with minimal computational overhead. On ImageNet at 80\% unstructured sparsity, REFLOW recovers ResNeXt-101 top-1 accuracy from below 0.41\% to 78.9\%, and at structured 2:4 N:M sparsity, it recovers ResNeXt-101 from 10.75\% to 79.07\%. By shifting the focus of the pruning paradigm from parameter selection to signal preservation, REFLOW delivers sparse models with state-of-the-art performance with minimal computational overhead.

Primary Area: optimization

Submission Number: 5229