A Region-Shrinking-Based Acceleration for Classification-Based Derivative-Free Optimization

Tianyi Han, Jingya LI, Zhipeng GUO, Yuan JIN

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Black-Box Optimization, Derivative-Free Optimization, Classification-Based Optimization

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Derivative-free optimization algorithms play an important role in scientific and engineering design optimization problems, especially when derivative information is not accessible. In this paper, we study the framework of classification-based derivative-free optimization algorithms. By introducing a concept called hypothesis-target shattering rate, we revisit the computational complexity upper bound of this type of algorithms. Inspired by the revisited upper bound, we propose an algorithm named “RACE-CARS”, which adds a random region-shrinking
step compared with “SRACOS” (Hu et al., 2017). We further establish a theorem showing the acceleration of region-shrinking. Experiments on the synthetic functions as well as black-box tuning for language-model-as-a-service demonstrate empirically the efficiency of “RACE-CARS”. An ablation experiment on the introduced hyperparameters is also conducted, revealing the mechanism of “RACE-CARS” and putting forward an empirical hyperparameter-tuning guidance.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3018