Revisiting Evolutionary Algorithms for Optimization for Deep Learning: Introducing DL-HEA: EAs for Optimization for Deep Learning

Toki Tahmid Inan, Amarda Shehu

Abstract: Despite the central role that optimization plays for deep learning in the model training phase, evolutionary-inspired approaches are largely under-represented in the literature. This paper proposes a Deep Learning Hybrid Evolutionary Algorithm (DL-HEA) which integrates a gradient-, mini batch-based local search operator within an evolutionary computing framework. DL-HEA shows competitive performance in optimization effectiveness and generalization capability. Findings suggest that hybrid evolutionary algorithms hold promise for addressing challenges posed by non-convex optimization for deep learning, offering a compelling alternative to Stochastic Gradient Descent in benchmarked settings and a way forward for novel optimization algorithms for deep learning.