NgIohTuned: a Highly Reproducible Black-box Optimization Wizard
Abstract: Inspired by the observations in neuro-control and various reproducibility issues in black-box optimization in the machine learning community, we revisit black-box optimization with rigorous benchmarking in mind. We (i) compare real-world (RW) benchmarks with artificial ones, emphasizing the success of Differential Evolution, Particle Swarm Optimization, and bet-and-run in the former case; (ii) introduce new artificial benchmarks, dubbed multi-scale benchmarks, with a focus on scaling issues (where scale refers to the unknown distance between the optimum and the origin), akin to real-world benchmarks such as those arising in neural reinforcement learning; (iii) demonstrate the performance of quasi-opposite sampling and of mathematical programming methods (Cobyla and direct search) on multi-scale continuous benchmarks; (iv) showcase the robustness and performance of algorithms focusing on a carefully chosen decreasing schedule of the mutation rate on discrete benchmarks; (v) design novel continuous black-box optimization strategies combining optimization algorithms with good scaling properties in the first phase, robust optimization techniques in the intermediate phase, and methods with fast convergence in the final optimization phase. Our methods are included in a public optimization wizard, available in two versions: NgIoh4 (which does not leverage information about the type of variables) and NgIohTuned (leveraging all conclusions of the present paper, including choosing algorithms thanks to high-level information on the real-world nature of a problem and/or its neuro-control nature and applying recent direct-search methods). They are integrated into a platform with complete reproducibility on a large benchmarking suite.
Submission Length: Long submission (more than 12 pages of main content)
Previous TMLR Submission Url: https://openreview.net/forum?id=0FDiCoIStW&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DTMLR%2FAuthors%23your-submissions)
Changes Since Last Submission: We made a major revision, fully described in the PDF enclosed as supplementary material.
Assigned Action Editor: ~Cedric_Archambeau1
Submission Number: 3330
Loading