GSparsity: Unifying Network Pruning and Neural Architecture Search by Group Sparsity

Avraam Chatzimichailidis, Arber Zela, Janis Keuper, Yang Yang

Abstract: In this paper, we propose a unified approach for network pruning and one-shot neural architecture search (NAS) via group sparsity. We first show that group sparsity via the recent Proximal Stochastic Gradient Descent (ProxSGD) algorithm achieves new state-of-the-art results for filter pruning. Then, we extend this approach to operation pruning, directly yielding a gradient-based NAS method based on group sparsity. Compared to existing gradient-based algorithms such as DARTS, the advantages of this new group sparsity approach are threefold. Firstly, instead of a costly bilevel optimization problem, we formulate the NAS problem as a single-level optimization problem, which can be optimally and efficiently solved using ProxSGD with convergence guarantees. Secondly, due to the operation-level sparsity, discretizing the network architecture by pruning less important operations can be safely done without any performance degradation. Thirdly, the proposed approach finds architectures that are both stable and well-performing on a variety of search spaces and datasets.

Keywords: deep learning, neural architecture search, pruning

One-sentence Summary: We propose a unified approach for network pruning and one-shot neural architecture search (NAS) via group sparsity

Track: Main track

Reproducibility Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Reviewers: Avraam Chatzimichailidis, [e-mail]

CPU Hours: 0

GPU Hours: 0

TPU Hours: 0

Datasets And Benchmarks: CIFAR-10, CIFAR-100, ImageNet, NAS-Bench-201