Meta-learning from sparse recovery

Beicheng Lou, Nathan Zhao, Jiahui Wang

Keywords: meta-learning, compressed sensing

TL;DR: high-order gradients can be safely ignored (bounded) for meta-learning in a regime where restricted isometry property is satisfied

Abstract:

Meta-learning aims to train a model on various tasks so that given sample data from a task, even if unforeseen, it can adapt fast and perform well.
We apply techniques from compressed sensing to shed light on the effect of inner-loop regularization in meta-learning, with an algorithm that minimizes cross-task interference without compromising weight-sharing.
In our algorithm, which is representative of numerous similar variations, the model is explicitly trained such that upon adding a pertinent sparse output layer, it can perform well on a new task with very few number of updates, where cross-task interference is minimized by the sparse recovery of output layer. We demonstrate that this approach produces good results on few-shot regression, classification and reinforcement learning, with several benefits in terms of training efficiency, stability and generalization.

Contribution Process Agreement: Yes

Author Revision Details:

We modified Figure 6 as suggested by the review.
We also added an explanation for Figure 5 as suggested by the review:
"Since meta-learning aims for the performance after a few steps of inner-loop tuning, there might be a regime where the performance of random initialization outperforms that of meta-network."
We modified Eq. 7 as suggested by the review.

Poster Session Selection: Poster session #3 (19:20 UTC+1)