Meta Learning in the Continuous Time Limit

Ruitu Xu, Lin Chen, amin karbasi

Abstract: In this paper, we establish the ordinary dif- ferential equation (ode) that underlies the training dynamics of Model-Agnostic Meta- Learning (maml). Our continuous-time limit view of the process eliminates the influence of the manually chosen step size of gradi- ent descent and includes the existing gradi- ent descent training algorithm as a special case that results from a specific discretiza- tion. We show that the maml ode enjoys a linear convergence rate to an approximate stationary point of the maml loss function for strongly convex task losses, even when the corresponding maml loss is non-convex. Moreover, through the analysis of the maml ode, we propose a new bi-maml training al- gorithm that reduces the computational bur- den associated with existing maml training methods, and empirical experiments are per- formed to showcase the superiority of our proposed methods in the rate of convergence with respect to the vanilla maml algorithm.