You Only Prune Once: A Zero-Shot, Data-Free Pruning at Initialization with Transferable Supermasks

ICLR 2026 Conference Submission18693 Authors

Keywords: Pruning at Initialization (PaI), zero-shot pruning, data-free pruning, transferable supermasks

TL;DR: We propose a zero-shot, data-free neural network pruning at initialization method that uses low-rank residuals to compute a once-only saliency, yielding transferable supermasks across datasets and sparsity budgets.

Abstract: Pruning at Initialization (PaI) accelerates training while maintaining accuracy, yet most criteria depend on data and backpropagation, leaving them brittle. Slight variations in random seed or sparsity budget reorder scores require re-scoring or iterative schedules and yield masks with weak transferability across seeds, datasets, and budgets. The proposed \emph{You Only Prune Once} (\textbf{YOPO}) framework addresses these limitations through a \emph{zero-shot}, data and gradient-free design. YOPO computes a \emph{once-only} saliency by fitting a nonnegative low-rank model to absolute weights at random initialization and measuring the element-wise Frobenius residual. Global or layer-wise thresholds generate masks with \emph{exact} sparsity control and no layer collapse. Since ordering and budget are decoupled, the same saliency supports \emph{re-thresholding} to any sparsity and \emph{dataset transfer} without re-scoring, enabling reusable "supermasks". Experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet with standard CNN backbones show that YOPO matches or surpasses strong single-shot PaI baselines, rivals iterative/data-dependent methods despite using no data at initialization, and consistently outperforms expander-graph zero-shot PaI. Altogether, YOPO provides a scalable and intuitive approach to initialization-time pruning with stable transfer across seeds, datasets, and sparsity levels.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 18693