Informative Robust Causal Representation for Generalizable Deep Learning

Mengyue Yang, Furui Liu, Xu Chen, Zhitang Chen, Jianye HAO, Jun Wang

Keywords: Causal Representation, Mutual Information, Robust Representation

Abstract: In many real-world scenarios, such as image classification and recommender systems, it is evidence that representation learning can improve model's performance over multiple downstream tasks. Existing learning approaches rely on establishing the correlation (or its proxy) between features and the downstream task (labels), which typically results in a representation containing cause, effect and spurious correlated variables of the label. Its generalizability may deteriorate because of the unstability of the non-causal parts. In this paper, we propose to learn causal representation from observational data by regularizing the learning procedure with mutual information measures according to our hypothetical causal graph. The optimization involves a counterfactual loss, based on which we deduce a theoretical guarantee that the causality-inspired learning is with reduced sample complexity and better generalization ability. Extensive experiments show that the models trained on causal representations learned by our approach is robust under adversarial attacks and distribution shift.

One-sentence Summary: A causality-inspired representation learning method by regularized mutual information based approach with guaranteed sample complexity reduction under certain assumptions.