Diversity-Aware Agnostic Ensemble of Sharpness Minimizers
Keywords: SAM, Ensemble learning, Sharpness-Aware Minimization
Abstract: There has long been a variety of theoretical and empirical evidence supporting the success of ensemble learning. Deep ensembles in particular leverage training randomness and expressivity of individual neural networks to gain prediction diversity and ultimately a boost in generalization performance, robustness and uncertainty estimation. In respect of generalization ability, it is found that minimizers pursuing wider local minima result in models being more robust to shifts between training and testing sets. A natural research question arises out of these two approaches as to whether better generalization ability can be achieved if ensemble learning and loss sharpness minimization is integrated. Our work takes the lead in investigating this connection and proposes DASH - a learning algorithm that promotes diversity and flatness within deep ensembles. More concretely, DASH encourages base learners to move divergently towards low-loss regions of minimal sharpness. We provide a theoretical backbone for our method along with empirical evidence demonstrating an improvement in ensemble generalization ability.
Primary Area: general machine learning (i.e., none of the above)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4200
Loading