Optimistic Meta-GradientsDownload PDF

Published: 21 Oct 2022, Last Modified: 26 Mar 2024NeurIPS 2022 Workshop MetaLearn PosterReaders: Everyone
Keywords: convex optimisation, optimism, meta-learning
TL;DR: We study the connection between gradient-based meta-learning and convex optimisation in the single task online meta-learning setting.
Abstract: We study the connection between gradient-based meta-learning and convex optimisation. We observe that gradient descent with momentum is as a special case of meta-gradients, and building on recent results in optimisation, we prove convergence rates for meta-learning in the single task setting. While a meta-learned update rule can yield faster convergence up to constant factor,it is not sufficient for acceleration. Instead, some form of optimism is required. We show that optimism in meta-learning can be captured through the recently proposed Bootstrapped Meta-Gradient method, providing deeper insight into its underlying mechanics.
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 1 code implementation](https://www.catalyzex.com/paper/arxiv:2301.03236/code)
0 Replies

Loading