A Half-Space Stochastic Projected Gradient Method for Group Sparsity Regularization

Tianyi Chen, Guanyi Wang, Tianyu DING, Bo Ji, Sheng Yi, Zhihui Zhu

Keywords: Group Sparsity, Stochastic Learning, Half-Space Projection, Group-Sparsity Identification

Abstract: Optimizing with group sparsity is significant in enhancing model interpretability in machining learning applications, e.g., feature selection, compressed sensing and model compression. However, for large-scale stochastic training problems, effective group-sparsity exploration are typically hard to achieve. Particularly, the state-of-the-art stochastic optimization algorithms usually generate merely dense solutions. To overcome this shortage, we propose a stochastic method—Half-space Stochastic Projected Gradient method (HSPG) to search solutions of high group sparsity while maintain the convergence. Initialized by a simple Prox-SG Step, the HSPG method relies on a novel Half-Space Step to substantially boosts the sparsity level. Numerically, HSPG demonstrates its superiority in deep neural networks, e.g., VGG16, ResNet18 and MobileNetV1, by computing solutions of higher group sparsity, competitive objective values and generalization accuracy.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We propose a Half-Space Projection to effectively explore group sparsity structure with theoretical convergence guarantee in order to overcome the limitation of existing stochastic proximal method.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=Dwu4UCM1a1