Vote for Nearest Neighbors Meta-Pruning of Self-Supervised Networks

Haiyan Zhao, Tianyi Zhou, Guodong Long, Jing Jiang, Liming Zhu, Chengqi Zhang

Keywords: network pruning, meta pruning, self-supervision, multi-task pruning

Abstract: Pruning plays an essential role in deploying deep neural nets (DNNs) to the hardware of limited memory or computation. However, current high-quality iterative pruning can create a terrible carbon footprint when compressing a large DNN for a wide variety of devices and tasks. Can we reuse the pruning results on previous tasks to accelerate the pruning for a new task? Can we find a better initialization for a new task, e.g., a much smaller network closer to the final pruned model, by exploiting its similar tasks? We study this ``nearest neighbors meta-pruning'' problem by first investigating different choices of pre-trained models for pruning under limited iterations. Our empirical study reveals several advantages of the self-supervision pre-trained model when pruned for multiple tasks. We further study the overlap of pruned models for similar tasks and how the overlap changes for different layers. Inspired by these discoveries, we develop a simple but strong baseline ``Meta-Vote Pruning (MVP)'' that significantly reduces the pruning iterations for a new task by initializing a sub-network from the pruned models for tasks similar to it. In experiments, we demonstrate the advantages of MVP by extensive empirical studies and comparisons with popular pruning methods.

One-sentence Summary: A simple yet effective ''nearest neighbors meta-pruning'' method accelerates the pruning for a new task using models pruned for other tasks based on a self-supervised pre-trained model .