Loss Adapted Plasticity: Learning From Data With Unreliable Sources

Alexander Capstick, Francesca Palermo, Payam Barnaghi

Keywords: Data Sources, Unreliable Data, Noisy Data, Noisy Labels

TL;DR: To learn from reliable and unreliable data sources, this paper demonstrates a technique that can be applied to any gradient descent optimiser: Update model weights as a function of the perceived reliability of data sources within a wider data set.

Abstract: When data is streaming from multiple sources, conventional training methods update model weights often assuming the same level of reliability for each source; that is: a model does not consider data quality of a specific source during training. In many applications, sources can have varied levels of noise or corruption that can produce negative effects on the learning of a robust machine learning model. A key issue is that the quality of data or labels for individual sources is often not available to a model during training and could vary over time. A solution to this problem is to consider the mistakes made while training on data originating from sources and utilise this to create a perceived data quality for each source. This paper demonstrates a technique that can be applied to any gradient descent optimiser: Update model weights as a function of the perceived reliability of data sources within a wider data set. The algorithm controls the plasticity of a given model to weight updates based on the history of losses from individual data sources. We show that applying this technique can significantly improve model performance when trained on a mixture of reliable and unreliable data sources, and maintain performance when models are trained on data sources that are all considered reliable.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning