Max-Affine Spline Insights Into Deep Network Pruning

Randall Balestriero, Haoran You, Zhihan Lu, Yutong Kou, Huihong Shi, Yingyan Lin, Richard Baraniuk

Keywords: DNN Interpretability, Network pruning, Max-affine spline theory, Visualization

Abstract: State-of-the-art (SOTA) approaches to deep network (DN) training overparametrize the model and then prune a posteriori to obtain a "winning ticket'' subnetwork that can be trained from scratch to achieve high accuracy. To date, the literature has remained largely empirical and hence provides little insights into how pruning affects a DN's decision boundary and no guidance regarding how to design a principled pruning technique. Using a recently developed spline interpretation of DNs, we develop new theory and visualization tools that provide new insights into how pruning DN nodes affects the decision boundary. We discover that a DN's spline mappings exhibit an early-bird (EB) phenomenon whereby the spline's partition converges at early training stages, bridging the recently developed max-affine spline theory and lottery ticket hypothesis of DNs. We leverage this new insight to develop a principled and efficient pruning strategy that focuses on a tiny fraction of DN nodes whose corresponding spline partition regions actually contribute to the final decision boundary. Extensive experiments on four networks and three datasets validate that our new spline-based DN pruning approach reduces training FLOPs by up to 3.5x while achieving similar or even better accuracy than state-of-the-art methods. All the codes will be released publicly upon acceptance.

One-sentence Summary: We leverage the spline theory to interpret network pruning and propose a principal and efficient pruning method.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2101.02338/code)