On Convergence Rates of Deep Nonparametric Regression under Covariate Shift

Chenguang Duan, Yuling Jiao, Lican Kang, Xiliang Lu, Jerry Zhijian Yang

Keywords: nonparametric regression, covariate shift, importance weighting, deep learning

Abstract:

Traditional machine learning and statistical modeling methodologies are rooted in a fundamental assumption: that both training and test data originate from the same underlying distribution. However, the practical reality often presents a challenge, as training and test distributions frequently manifest discrepancies or biases. In this work, we study covariate shift, a type of distribution mismatches, in the context of deep nonparametric regression. We thus formulate a two-stage pre-training reweighted framework relying on deep ReLU neural networks. We rigorously establish convergence rates for the unweighted, reweighted, and pre-training reweighted estimators, illuminating the pivotal role played by the density-ratio reweighting strategy. Additionally, our analysis illustrates the efficacy of pre-training and provides valuable insights for practitioners, offering guidance for the judicious selection of the number of pre-training samples.

Primary Area: learning theory

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1703