Do Symbolic or Black-Box Representations Generalise Better In Learned Optimisation?
Keywords: reinforcement learning, meta learning, optimisation
TL;DR: We explore the difference in generalisation capabilities between black-box learned optimisers, which are very common, and discovered symbolic optimisers, which are less typical.
Abstract: Until recently, behind every algorithmic advance in machine learning was a human researcher. Now, however, algorithms can be meta-learned automatically, with little human input. However, to be truly useful, such algorithms must generalise beyond their training distribution. This is especially challenging in reinforcement learning (RL), where transferring algorithms between environments with vastly different dynamics is difficult and training on diverse environments often requires prohibitively expensive large-scale data collection.Learned optimisation is a branch of algorithmic discovery that meta-learns optimiser update rules. Learned optimisers can be classified into two groups: black-box algorithms, where the optimiser is a neural network; or symbolic algorithms, where the optimiser is represented using mathematical functions or code. While some claim that symbolic algorithms generalise better than black-box ones, testing such assertions is complicated by the fact that symbolic algorithms typically include additional hyperparameters, and thus their evaluation is done many-shot. This is an unfair comparison with the zero-shot evaluation of black-box optimisers. In this work, we build a pipeline to discover symbolic optimisers which are hyperparameter-free, enabling a fair comparison of the generalisation of symbolic optimisers with that of an open-source state-of-the-art black-box optimiser trained for RL. Based on our analysis, we propose suggestions to improve the symbolic optimiser discovery pipeline for RL, with an overall objective of reducing the need for hyperparameter tuning to train an agent.
Primary Area: reinforcement learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3838
Loading