HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning

TMLR Paper4943 Authors

Abstract: Structured pruning is a popular technique for compressing deep neural networks (DNNs) into efficient sub-networks. However, existing methods often require multi-stage process, engineering efforts, and human expertise. The Only-Train-Once series (OTOv1-v3) has been proposed to resolve some pain points by streamlining the workflow. However, the built-in sparse optimizers in the OTO series need hyperparameter tuning and implicit control over sparsity, necessitating human intervention. To address these limitations, we propose the Hybrid Efficient Structured Sparse Optimizer (HESSO), which automatically and efficiently train a DNN within a single run to produce a high-performing sub-network. HESSO is almost tuning-free and enjoys user-friendly integration for generic training applications. In addition, to tackle the common issue of irreversible pruning performance collapse in certain DNNs, we further propose the Corrective Redundant Identification Cycle (CRIC), which integrates seamlessly with HESSO. The extensive numerical results showcase that HESSO can achieve competitive performance on various state-of-the-art benchmarks and support most DNN architectures. Moreover, CRIC can effectively prevent the irreversible performance collapse and further enhance the performance of HESSO on certain applications.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Yani_Ioannou1

Submission Number: 4943