Fight Fire with Fire: Multi-biased Interactions in Hard-Thresholding

Xinzhe Yuan, Xiang Peng, Huan Xiong, Bin Gu

Keywords: Optimzation, Biased Gardient, Zeroth-Order, Hard-Thresholding

Abstract: $\ell_0$ constrained optimization is widely used in machine learning, especially for high-dimensional problems, as it effectively promotes sparse learning. A prominent technique for solving these problems is Hard-Thresholding gradient descent. However, the inherent expansibility of Hard-Thresholding operators can lead to convergence issues, necessitating strategies to accelerate the algorithm. In this article, we believe the random Hard-Thresholding algorithm can be interpreted as an equivalent biased gradient algorithm. By introducing appropriate biases, we can mitigate some of the issues of Hard-Thresholding and enhance convergence. We categorize the biases into memory-biased and recursively-biased, examining their distinct applications within Hard-Thresholding algorithms. Next, we explore the Zeroth-Order versions of these algorithms, which introduce additional biases from Zeroth-Order gradients. Our findings indicate that recursively bias effectively counteracts some of the issues caused by Hard-Thresholding, resulting in improved performance for First-Order algorithms. Conversely, due to the accumulation of errors from Zeroth-Order gradients during recursive bias, the performance of Zeroth-Order algorithms is inferior to that influenced by historical gradients. To address these insights, we propose the SARAHT and BVR-SZHT algorithms for First-Order and Zeroth-Order Hard-Thresholding, respectively, both of which demonstrate faster convergence speeds compared to previous methods. We validate our hypotheses through black-box adversarial experiments and ridge regression evaluations.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9189