POPULATION DESCENT: A NATURAL-SELECTION BASED HYPER-PARAMETER TUNING FRAMEWORK
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: hyperparameter tuning, memetic algorithm, differential optimization, Gaussian mutations, supervised learning
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: First-order gradient descent has been the base of the most successful optimization algorithms ever implemented.
On supervised learning problems with very high dimensionality, such as neural network optimization, it is almost always the algorithm of choice, mainly due to its memory and computational efficiency.
However, it is a classical result in optimization that gradient descent converges to local minima on non-convex functions. Even more importantly, in certain high-dimensional cases, escaping the plateaus of large saddle points becomes intractable.
On the other hand, black-box optimization methods are not sensitive to the local structure of a loss function's landscape but suffer the curse of dimensionality.
Instead, memetic algorithms aim to combine the benefits of both.
Inspired by this, we present Population Descent, a memetic algorithm focused on hyperparameter optimization.
We show that an adaptive $m$-elitist selection approach combined with a normalized-fitness-based randomization scheme outperforms more complex state-of-the-art algorithms by up to 13\% on common benchmark tasks.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6683
Loading