DFMTDS: Distribution-free martingale test Distribution shift

Anonymous

Keywords: Conformal prediction, Martingales, Distribution shift detection, Distribution-free testing, Transductive inference

Abstract: A standard assumption in machine learning is the data are generated by a fixed but unknown probability, which is equivalent to assuming the examples are generated from the same probability distribution independently. So for most of the learning research we usually randomly shuffle the whole data into training and test data set. However, for real-life application reality is that the data points are observed one by one. This paper is devoted to testing the assumption of distribution shift on-line: the observed data arrive one by one, and after receiving each object, the machine learning algorithms give a prediction label, we would like to have a valid measure of the degree to which the evidence to against the assumption of non-distribution-shift. Such measures are provided under the framework of distribution-free methods, also called martingales measure, which is a general empirical theory of probability developed in 1993-2003. We report the experimental performance of martingales to measure on the real-life data sets and the results to show a bona fide fact that the distribution shift testing is an inescapable reality when we adaptive machine learning algorithms to the original order.