Optimization over Sparse Support-Preserving Sets: Two-Step Projection with Global Optimality Guarantees

William de Vazelhes, Xiaotong Yuan, Bin Gu

TL;DR: We simplify and extend convergence proofs for Iterative Hard-Thresholding using a non-convex three-point lemma, improving existing results in standard settings and providing new results under additional constraints.

Abstract: In sparse optimization, enforcing hard constraints using the $\ell_0$ pseudo-norm offers advantages like controlled sparsity compared to convex relaxations. However, many real-world applications demand not only sparsity constraints but also some extra constraints. While prior algorithms have been developed to address this complex scenario with mixed combinatorial and convex constraints, they typically require the closed form projection onto the mixed constraints which might not exist, and/or only provide local guarantees of convergence which is different from the global guarantees commonly sought in sparse optimization. To fill this gap, in this paper, we study the problem of sparse optimization with extra *support-preserving* constraints commonly encountered in the literature. We present a new variant of iterative hard-thresholding algorithm equipped with a two-step consecutive projection operator customized for these mixed constraints, serving as a simple alternative to the Euclidean projection onto the mixed constraint. By introducing a novel trade-off between sparsity relaxation and sub-optimality, we provide global guarantees in objective value for the output of our algorithm, in the deterministic, stochastic, and zeroth-order settings, under the conventional restricted strong-convexity/smoothness assumptions. As a fundamental contribution in proof techniques, we develop a novel extension of the classic three-point lemma to the considered two-step non-convex projection operator, which allows us to analyze the convergence in objective value in an elegant way that has not been possible with existing techniques. In the zeroth-order case, such technique also improves upon the state-of-the-art result from de Vazelhes et. al. (2022), even in the case without additional constraints, by allowing us to remove a non-vanishing system error present in their work.

Lay Summary:

Many real-world problems, from sensor selection to feature selection in machine learning, involve picking just a few important variables while meeting additional conditions. This is called “sparse optimization with constraints.” Solving such problems can be tricky, especially when the usual mathematical tools don’t work well due to complex requirements.

Our work introduces a new algorithm that can handle these situations more effectively. It builds on a popular method called iterative hard thresholding but adapts it to deal with both sparsity and additional constraints by breaking the problem into two simple steps.
We also developed simpler and more extensible mathematical tool to help analyze how well the algorithm performs, improving on previous results.

This work helps make powerful optimization techniques more widely usable in practical, real-world situations.

Link To Code: https://github.com/wdevazelhes/2SP_icml2025

Primary Area: Optimization

Keywords: optimization, sparse optimization, iterative hard-thresholding, three-point lemma, approximate global guarantees, projected gradient descent, stochastic gradient descent, zeroth-order

Submission Number: 6044