Estimation of prediction error with known covariate shift

Hui Xu, Robert Tibshirani

Keywords: covariate shift, out-of-distribution error

TL;DR: We consider estimation of generalization error when the covariate shift between training and test data is observed, and propose a method based on parametric bootstrap by leveraging the additional covariate information of test data.

Abstract: In supervised learning, the estimation of prediction error on unlabeled test data is an important task. Existing methods are usually built on the assumption that the training and test data are sampled from the same distribution, which is often violated in practice. As a result, traditional estimators like cross-validation (CV) will be biased and this may result in poor model selection. In this paper, we assume that we have a test dataset in which the feature values are available but not the outcome labels, and focus on a particular form of distributional shift of covariate shift. We propose an alternative method based on parametric bootstrap of the target of conditional error ErrX. Empirically our method outperforms CV for both simulation and real data example across different modeling tasks, and is comparable to state-of-the-art methods for image classification.