R-divergence for Estimating Model-oriented Distribution Discrepancy

Zhilin Zhao, Longbing Cao

Keywords: non-IID, Distribution Discrepancy, Data Divergence, Two-sample Test

TL;DR: R-divergence is proposed to estimate the discrepancy between two probability distributions for specific supervised or unsupervised machine learning models.

Abstract: Real-life data are often non-IID due to complex distributions and interactions, and the sensitivity to the distribution of samples can differ among learning models. Accordingly, a key question for any supervised or unsupervised model is whether the probability distributions of two given datasets can be considered identical. To address this question, we introduce R-divergence, designed to assess model-oriented distribution discrepancies. The core insight is that two distributions are likely identical if their optimal hypothesis yields the same expected risk for each distribution. To estimate the distribution discrepancy between two datasets, R-divergence learns a minimum hypothesis on the mixed data and then gauges the empirical risk difference between them. We evaluate the test power across various unsupervised and supervised tasks and find that R-divergence achieves state-of-the-art performance. To demonstrate the practicality of R-divergence, we employ R-divergence to train robust neural networks on samples with noisy labels.

Submission Number: 1840