Knowledge-Adaptation Priors

Mohammad Emtiyaz Khan, Siddharth Swaroop

Keywords: Adaptation, Bayesian methods, Priors, Online learning

TL;DR: We propose a new prior which enables quick and accurate adaptation for a wide-variety of tasks and models.

Abstract: Humans and animals have a natural ability to quickly adapt to their surroundings, but machine-learning models, when subjected to changes, often require a complete retraining from scratch. We present Knowledge-adaptation priors (K-priors) to reduce the cost of retraining by enabling quick and accurate adaptation for a wide-variety of tasks and models. This is made possible by a combination of weight and function-space priors to reconstruct the gradients of the past, which recovers and generalizes many existing, but seemingly-unrelated, adaptation strategies. Training with simple first-order gradient methods can often recover the exact retrained model to an arbitrary accuracy by choosing a sufficiently large memory of the past data. Empirical results show that adaptation with K-priors achieves performance similar to full retraining, but only requires training on a handful of past examples.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/team-approx-bayes/kpriors

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2106.08769/code)