Curvature-Aware Derivative-Free Optimization

Bumsu Kim, Daniel McKenzie, HanQin Cai, Wotao Yin

Abstract: The paper discusses derivative-free optimization (DFO), which involves minimizing a function without access to gradients or directional derivatives, only function evaluations. Classical DFO methods such as Nelder-Mead and direct search have limited scalability for high-dimensional problems. Zeroth-order methods, which mimic gradient-based methods, have been gaining popularity due to the demands of large-scale machine learning applications. This paper focuses on the selection of the step size \(\alpha _k\) in such methods. The proposed approach, called Curvature-Aware Random Search (CARS), uses first- and second-order finite difference approximations to compute a candidate \(\alpha _{+}\). A safeguarding step then evaluates \(\alpha _{+}\) and chooses an alternate step size in case \(\alpha _{+}\) does not decrease the objective function. We prove that for strongly convex objective functions, CARS converges linearly provided that the search direction is drawn from a distribution satisfying very mild conditions. We also present a Cubic Regularized variant of CARS, named CARS-CR, which provably converges at a rate of \(\mathcal {O}(1/k)\) without the assumption of strong convexity. Numerical experiments show that CARS and CARS-CR match or exceed the state-of-the-art on benchmark problem sets.