Meta-learning with differentiable closed-form solvers

Luca Bertinetto, Joao F. Henriques, Philip Torr, Andrea Vedaldi

Abstract:

Adapting deep networks to new concepts from a few examples is challenging, due to the high computational requirements of standard fine-tuning procedures.
Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent.
Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efficiently.
In this paper, we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning.
The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data.
This requires back-propagating errors through the solver steps.
While normally the cost of the matrix operations involved in such a process would be significant, by using the Woodbury identity we can make the small number of examples work to our advantage.
We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components.
Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks.

Keywords: few-shot learning, one-shot learning, meta-learning, deep learning, ridge regression, classification

TL;DR: We propose a meta-learning approach for few-shot classification that achieves strong performance at high-speed by back-propagating through the solution of fast solvers, such as ridge regression or logistic regression.

Code: [ 5 community implementations](https://paperswithcode.com/paper/?openreview=HyxnZh0ct7)

Data: [CIFAR-FS](https://paperswithcode.com/dataset/cifar-fs), [CIFAR-100](https://paperswithcode.com/dataset/cifar-100), [mini-Imagenet](https://paperswithcode.com/dataset/mini-imagenet)

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:1805.08136/code)