Max-margin Inspired Per-sample Re-weighting for Robust Deep Learning

Ramnath Kumar, Kushal Alpesh Majmundar, Dheeraj Mysore Nagaraj, Arun Suggala

Keywords: Deep Learning, Robustness, Per-sample weighting

TL;DR: We propose a new per-sample weighting scheme that shows improvement on various robustness tasks

Abstract:

We design simple, explicit, and flexible per-sample re-weighting schemes for learning deep neural networks in a variety of tasks that require robustness of some form. These tasks include classification with label imbalance, domain adaptation, and tabular representation learning. Our re-weighting schemes are simple and can be used in combination with any popular optimization algorithms such as SGD, Adam.
Our techniques are inspired by max-margin learning, and rely on mirror maps such as log-barrier and negative entropy, which have been shown to perform max-margin classification. Empirically, we demonstrate the superiority of our approach on all of the aforementioned tasks. Our techniques provide state-of-the-art results in tasks involving tabular representation learning and domain adaptation.