Experimental Design for Nonstationary Optimization
Keywords: plasticity, continual learning, experiment design
TL;DR: We conduct an empirical analysis of the methods and experimental design decisions prominent in plasticity research.
Abstract: Traditional methods for optimizing neural networks often struggle when used
to train networks in settings where the data distributions change, and plasticity
preservation methods have been shown to improve performance in such settings
(e.g. continual learning and reinforcement learning). With the growing inter-
est in nonstationary optimization and plasticity research, there is also a growing
need to properly define experimental design and hyperparameter search protocols
to enable principled research. Each new proposed work typically adds several
new hyperparameters makes many more design decisions such as hyperparame-
ter selection protocols, evaluation protocols, and types of tasks examined. While
innovation in experiment design is important, it is also necessary to (1) question
whether those innovations are leading to the best progress and (2) have standard-
ized practices that make it easier to directly compare to prior works. In this paper,
we first perform an extensive empirical study of over 27,000 trials looking at the
performance of different methods and hyperparameters across different settings
and architectures used in the literature to provide an evaluation of these methods
and the hyperparameters they use under similar experimental conditions. We then
examine several core experiment design choices made by the community, affirm-
ing some while providing evidence against others, and provide concrete recom-
mendations and analysis that can be used to guide future research.
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 10996
Loading