Linearly Independent Feature Extraction

Mohamed Ishmael Belghazi

Keywords: Representation learning, Out-of-domain generalization, information theory, spectral methods

TL;DR: Representations with controllable dependence on the state of the world using an eigensolver.

Abstract: We argue that domain invariance is fundamentally limiting as an objective for out-of-domain generalization (OOD) and propose a more nuanced alternative: Modeling a full spectrum of dependence on the state of the world. We make this concept tractable by developing a spectral theory, grounded in a novel operator algebra, that is formally equivalent to information-theoretic measures of dependence. The culmination is Linearly Independent Feature Extraction (LIFE): An algorithm for learning representations with controllable state-dependence, implemented using a simple eigensolver. Analytical evaluation on known data-generating processes demonstrates that LIFE recovers oracle-level features. Empirically, on linear hypothesis LIFE outperforms current gold standards and, on some datasets, even surpasses deep invariant models. A broadly applicable dynamic theory of state-dependence emerges.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 2399