Fundamental Limits and Tradeoffs in Invariant Representation Learning

Han Zhao, Chen Dan, Bryon Aragam, Tommi S. Jaakkola, Geoff Gordon, Pradeep Kumar Ravikumar

Keywords: Representation learning

Abstract: Many machine learning applications involve learning representations that achieve two competing goals: To maximize information or accuracy with respect to a target while simultaneously maximizing invariance or independence with respect to a subset of features. Typical examples include privacy-preserving learning, domain adaptation, and algorithmic fairness, just to name a few. In fact, all of the above problems admit a common minimax game-theoretic formulation, whose equilibrium represents a fundamental tradeoff between accuracy and invariance. In this paper, we provide an information-theoretic analysis of this general and important problem under both classification and regression settings. In both cases, we analyze the inherent tradeoffs between accuracy and invariance by providing a geometric characterization of the feasible region in the information plane, where we connect the geometric properties of this feasible region to the fundamental limitations of the tradeoff problem. In the regression setting, we also derive a tight lower bound on the Lagrangian objective that quantifies the tradeoff between accuracy and invariance. Our results shed new light on this fundamental problem by providing insights on the interplay between accuracy and invariance. These results deepen our understanding of this fundamental problem and may be useful in guiding the design of adversarial representation learning algorithms.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We provide an information-theoretic analysis of the tradeoff problem between accuracy and invariance under both classification and regression settings.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2012.10713/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=SuDXzSp7up