Adaptive backtracking for fast optimization
Keywords: Optimization, backtracking line search, Armijo condition, descent lemma, adaptive optimization methods
TL;DR: We propose a way to make backtracking line search subroutines adaptive by adjusting step sizes using a variable factor that depends on the criterion enforced by the line search.
Abstract: Backtracking line search is foundational in numerical optimization.
The basic idea is to adjust the step size of an algorithm by a {\em constant} factor until some chosen criterion (e.g. Armijo, Goldstein, Descent Lemma) is satisfied.
We propose a new way for adjusting step sizes, replacing the constant factor used in regular backtracking with one that takes into account the degree to which the chosen criterion is violated, without additional computational burden.
We perform a variety of experiments on over fifteen real world datasets, which confirm that adaptive backtracking often leads to significantly faster optimization.
For convex problems, we prove adaptive backtracking requires fewer adjustments to produce a feasible step size than regular backtracking does for two popular line search criteria: the Armijo condition and the descent lemma.
For nonconvex smooth problems, we prove adaptive backtracking enjoys the same guarantees of regular backtracking.
Primary Area: optimization
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 7304
Loading