Combating Spurious Features by Distribution Difference Regularization

Anonymous

Abstract: Prior studies show that spurious features are inevitable to avoid in the data collection process. These spurious features cause a shortcut for a model making bad prediction in real world test data due to ignoring the real features. In this work, we focus ondesigning a learning scheme to hinder the model from leveraging spurious features. To achieve this, prior studies usually make strong assumptions about the spurious features and identify them purely by manipulating the training data. In contrast, we make weaker assumptions and purpose a new framework for combating spurious features by observing the distribution shift between training and auxiliary data. In particular, with the help of unlabeled auxiliary data, we design a regularization technique based on the embedding distribution difference between training and auxiliary data to mitigate the effect of spurious features. Experimental results on NLI and coreference resolution tasks demonstrate that we improve the models on out-of-domain test data and reduce the contribution of spurious features in model predictions.