Variance Based Sample Weighting for Supervised Deep Learning

Paul Novello, Gaël Poëtte, David Lugato, Pietro Congedo

Keywords: supervised learning, sample distribution, statistical methods, sample weighting, approximation theory, Taylor expansion

Abstract: In the context of supervised learning of a function by a Neural Network (NN), we claim and empirically justify that a NN yields better results when the distribution of the data set focuses on regions where the function to learn is steeper. We first traduce this assumption in a mathematically workable way using Taylor expansion. Then, theoretical derivations allow to construct a methodology that we call Variance Based Samples Weighting (VBSW). VBSW uses local variance of the labels to weight the training points. This methodology is general, scalable, cost effective, and significantly increases the performances of a large class of models for various classification and regression tasks on image, text and multivariate data. We highlight its benefits with experiments involving NNs from shallow linear NN to ResNet or Bert.

One-sentence Summary: This paper constructs a new training distribution to weight the training data set and boost the performances of a Neural Network.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=NW6u1r3JGX