Denoising Low-Rank Data Under Distribution Shift: Double Descent and Data Augmentation

Chinmaya Kausik, Kashvi Srivastava, Rishi Sonthalia

Keywords: Denoising, Low Rank, distribution shift, double descent, overfitting, supervised, benign overfitting, autoencoder, covariate shift, out of distribution generalization

TL;DR: We compute the test error for linear denoisers for low-rank data in the presence of covariate shifts, experimentally verify our results, and derive insights about double descent, data augmentation and benign overfitting from them.

Abstract:

Despite the importance of denoising in modern machine learning and ample empirical work on supervised denoising, its theoretical understanding is still relatively scarce. One concern about studying supervised denoising is that one might not always have noiseless training data from the test distribution. It is more reasonable to have access to noiseless training data from a different dataset than the test dataset. Motivated by this, we study supervised denoising and noisy-input regression under distribution shift. We add three considerations to increase the applicability of our theoretical insights to real-life data and modern machine learning. First, while most past theoretical work assumes that the data covariance matrix is full-rank and well-conditioned, empirical studies have shown that real-life data is approximately low-rank. Thus, we assume that our data matrices are low-rank. Second, we drop independence assumptions on our data. Third, the rise in computational power and dimensionality of data have made it important to study non-classical regimes of learning. Thus, we work in the non-classical proportional regime, where data dimension $d$ and number of samples $N$ grow as $d/N = c + o(1)$.

For this setting, we derive general test error expressions for both denoising and noisy-input regression, and study when overfitting the noise is benign, tempered or catastrophic. We show that the test error exhibits double descent under general distribution shift, providing insights for data augmentation and the role of noise as an implicit regularizer. We also perform experiments using real-life data, where we match the theoretical predictions with under 1\% MSE error for low-rank data.

Submission Number: 51