Deep Learning Optimization Tasks and Metaheuristic Methods

Rafal Biedrzycki, Pawel Zawistowski, Bartlomiej Twardowski

Abstract: In this paper we identify and formulate two optimization tasks solved in connection with training DL models and constructing adversarial examples. This guides our review of optimization methods commonly used within the DL community. Simultaneously, we present findings from the literature concerning metaheuristics and black-box optimization. We focus on well-known optimizers suitable for solving ℝN tasks, which achieve good results on benchmarks and in competitions. Finally, we look into the research connected with utilizing metaheuristic optimization methods in combination with deep learning models.