Dataset Meta-Learning from Kernel Ridge-Regression

Timothy Nguyen, Zhourong Chen, Jaehoon Lee

Keywords: dataset distillation, dataset compression, meta-learning, kernel-ridge regression, neural kernels, infinite-width networks, dataset corruption

Abstract:

One of the most fundamental aspects of any machine learning algorithm is the training data used by the algorithm.
We introduce the novel concept of $\epsilon$-approximation of datasets, obtaining datasets which are much smaller than or are significant corruptions of the original training data while maintaining similar performance. We introduce a meta-learning algorithm Kernel Inducing Points (KIP) for obtaining such remarkable datasets, drawing inspiration from recent developments in the correspondence between infinitely-wide neural networks and kernel ridge-regression (KRR). For KRR tasks, we demonstrate that KIP can compress datasets by one or two orders of magnitude, significantly improving previous dataset distillation and subset selection methods while obtaining state of the art results for MNIST and CIFAR10 classification. Furthermore, our KIP-learned datasets are transferable to the training of finite-width neural networks even beyond the lazy-training regime. Consequently, we obtain state of the art results for neural network dataset distillation with potential applications to privacy-preservation.

One-sentence Summary: We introduce a meta-learning approach to distilling datasets, achieving state of the art performance for kernel-ridge regression and neural networks.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Data: [MNIST](https://paperswithcode.com/dataset/mnist)

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2011.00050/code)