AMSC: Adaptive Multi-Dimensional Structured Compression with Theoretical Guarantees
Keywords: Multi-dimensional structured compression, adaptive group lasso, selection consistency
Abstract: Network pruning is a pivotal strategy for reducing complexity and accelerating inference. Most pruning methods focus on a single dimension (depth or width), leading to insufficient compression when multiple dimensions are redundant. Additionally, separating pruning from training disrupts established network correlations, causing performance degradation. In this paper, we propose a novel Adaptive Multi-dimensional Structured Compression (AMSC) method that simultaneously learns the minimal depth, the minimal width, and network parameters under the strategy that prioritizes depth compression. Specifically, based on the regularization technique, AMSC incorporates layer- and filter- specific information into the penalty in order to adaptively identify and eliminate redundant depth and width in terms of the importance and size of each layer and filter. It integrates compression and training processes together without pruning. Consequently, the proposed method enables adaptive structure reduction from the initial configuration to a structure necessary that minimizes the generalization error. Rigorous theoretical evidence is provided in terms of the consistency of AMSC in achieving minimal network depth and width. To the best of our knowledge, this is the first study that offers a theoretical guarantees in structure selection. Extensive experiments on CIFAR-10/100 and ImageNet datasets demonstrate our method not only achieves state-of-the-art compression performance in terms of FLOPs and total parameters, but also preserves competitive classification accuracy. For example, AMSC enhances the accuracy of ResNet56 on CIFAR-10 from 93.37\% to 93.71\%, while simultaneously reducing calculations by 58.63\% and parameters by 44.71\%.
Supplementary Material: zip
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4329
Loading