Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression

Vincent Mai, Waleed Khamies, Liam Paull

Keywords: Regression, Noisy labels, Supervised Learning, Uncertainty, Variance, Heteroscedastic, Privileged Information

Abstract: In model learning, when the training dataset on which the parameters are optimized and the testing dataset on which the model is evaluated are not sampled from identical distributions, we say that the datasets are misaligned. It is well-known that this misalignment can negatively impact model performance. A common source of misalignment is that the inputs are sampled from different distributions. Another source for this misalignment is that the label generating process used to create the training dataset is imperfect. In this work, we consider this setting and additionally assume that the label generating process is able to provide us with a quantity for the role of each label in the misalignment between the datasets, which we consider to be privileged information. Specifically, we consider the task of regression with labels corrupted by heteroscedastic noise and we assume that we have access to an estimate of the variance over each sample. We propose a general approach to include this privileged information in the loss function together with dataset statistics inferred from the mini-batch to mitigate the impact of the dataset misalignment. Subsequently, we propose a specific algorithm for the heteroscedastic regression case, called Batch Inverse-Variance weighting, which adapts inverse-variance weighting for linear regression to the case of neural network function approximation. We demonstrate that this approach achieves a significant improvement in network training performances compared to baselines when confronted with high, input-independent noise.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: A method to reduce the effect of heteroscedastic noisy labels in regression by weighting them based on their variance and the variance of the other samples in the minibatch.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2107.04497/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=YWSmhmSmXc