Continual Nonlinear ICA-Based Representation Learning

Boyang Sun, Ignavier Ng, Guangyi Chen, Yifan Shen, Qirong Ho, Kun Zhang

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Casual Representation Learning, Nonlinear ICA, Continual Learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We develop a novel approach for nonlinear ICA that effectively accommodates continually arriving domains and show its identifiability from subspace to component-wise level.

Abstract: Unsupervised identification of disentangled representations remains a challenging problem. Recent progress in nonlinear Independent Component Analysis (ICA) provides a promising causal representation learning framework by separating latent sources from observable nonlinear mixtures. However, its identifiability hinges on the incorporation of side information, such as time or domain indexes, which are challenging to obtain adequately offline in real-world scenarios. In this paper, we develop a novel approach for nonlinear ICA that effectively accommodates continually arriving domains. We first theoretically demonstrate that model identifiability escalates from subspace to component-wise identifiability as new domains are involved. It motivates us to maintain prior knowledge and progressively refine it using new arriving domains. Upon observing a new domain, our approach optimizes the model by satisfying two objectives: (1) reconstructing the observations within the current domain, and (2) preserving the reconstruction capabilities for prior domains through gradient constraints. Experiments demonstrate that our method achieves performance comparable to nonlinear ICA methods trained jointly on multiple offline domains, demonstrating its practical applicability in continual learning scenarios.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4487