Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective

Can Jin, Tianjin Huang, Yihua Zhang, Mykola Pechenizkiy, Sijia Liu, Shiwei Liu, Tianlong Chen

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: neural network sparsification, visual prompt

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: This paper conducts systematic investigations about the impact of different visual prompts on model pruning and suggests an effective joint optimization approach.

Abstract: The rapid development of large-scale deep learning models questions the affordability of hardware platforms, which necessitates the pruning to reduce their computational and memory footprints. Sparse neural networks as the product, have demonstrated numerous favorable benefits like low complexity, undamaged generalization, $\textit{etc}$. Most of the prominent pruning strategies are invented from a $\textit{model-centric}$ perspective, focusing on searching and preserving crucial weights by analyzing network topologies. However, the role of data and its interplay with model-centric pruning has remained relatively unexplored. In this research, we introduce a novel $\textit{data-model co-design}$ perspective: to promote superior weight sparsity by learning important model topology and adequate input data in a synergetic manner. Specifically, customized $\textbf{V}$isual $\textbf{P}$rompts are mounted to upgrade neural $\textbf{N}$etwork $\textbf{s}$parsification in our proposed $\textbf{\texttt{VPNs}}$ framework. As a pioneering effort, this paper conducts systematic investigations about the impact of different visual prompts on model pruning and suggests an effective joint optimization approach. Extensive experiments with $3$ network architectures and $8$ datasets evidence the substantial performance improvements from $\textbf{\texttt{VPNs}}$ over existing start-of-the-art pruning algorithms. Furthermore, we find that subnetworks discovered by $\textbf{\texttt{VPNs}}$ from pre-trained models enjoy better transferability across diverse downstream scenarios. These insights shed light on new promising possibilities of data-model co-designs for vision model sparsification. Codes are in the supplement.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2675