Sample-Efficient Pruning Model Selection via Lasso

Kyuwook Chai, Taehyun Hwang, Min-hwan Oh

Keywords: model selection, neural network pruning, generalization error, sample complexity

TL;DR: We propose a novel elimination-based algorithm that identifies a near-optimal pruned network from a pool of pruned neural networks.

Abstract: We study the problem of selecting a pruned neural network from a set of candidates generated by various pruning methods. The goal of a learner is to identify a near-optimal model that achieves low generalization error. Although model selection techniques such as cross-validation are widely used in practice, they often fail to provide guarantees on generalization error or offer only asymptotic guarantees. To address these limitations, we propose an algorithm that jointly selects a pruned network and updates its parameters using an $L_1$-regularization, thereby encouraging sparsity while ensuring low generalization error. For a given error tolerance $\epsilon$, we establish a sample complexity lower bound of $\Omega\left(\frac{1}{\epsilon^2} \log M\right)$, where $M$ is the number of candidate models, demonstrating that our algorithm remains sample-efficient even when the candidate pool is large. Extensive numerical experiments confirm both the practical effectiveness and the theoretical guarantees of the proposed method.

Primary Area: learning theory

Submission Number: 6503