Meta Two-Sample Testing: Learning Kernels for Testing with Limited Data

Feng Liu, Wenkai Xu, Jie Lu, Danica J. Sutherland

Keywords: distinguishing distributions, statistical testing, meta-learning, deep kernel methods

TL;DR: Using related data in meta-learning can lead to far more powerful two-sample tests on small sample sizes.

Abstract: Modern kernel-based two-sample tests have shown great success in distinguishing complex, high-dimensional distributions by learning appropriate kernels (or, as a special case, classifiers). Previous work, however, has assumed that many samples are observed from both of the distributions being distinguished. In realistic scenarios with very limited numbers of data samples, it can be challenging to identify a kernel powerful enough to distinguish complex distributions. We address this issue by introducing the problem of meta two-sample testing (M2ST), which aims to exploit (abundant) auxiliary data on related tasks to find an algorithm that can quickly identify a powerful test on new target tasks. We propose two specific algorithms for this task: a generic scheme which improves over baselines, and a more tailored approach which performs even better. We provide both theoretical justification and empirical evidence that our proposed meta-testing schemes outperform learning kernel-based tests directly from scarce observations, and identify when such schemes will be successful.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/fengliu90/MetaTesting

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:2106.07636/code)