Beyond Disentanglement: On the Orthogonality of Learned Representations
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Disentanglement, Orthogonality, Unsupervised Learning, Representation Learning, DCI, DCI-ES
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: Proposing Importance-Weighted Orthogonality (IWO) as an alternative to traditional disentanglement metrics, we evaluate orthogonality in learned representations, demonstrating enhanced correlation with downstream task performance across applications.
Abstract: Evaluating learned representations independently of designated downstream tasks is pivotal for crafting robust and adaptable algorithms across a diverse array of applications. Among such evaluations, the assessment of disentanglement in a learned representation has emerged as a significant technique. In a disentangled representation, independent data generating factors are encoded in mutually orthogonal subspaces, a characteristic enhancing numerous downstream applications, potentially bolstering interpretability, fairness, and robustness. However, a representation is often deemed well-disentangled if these orthogonal subspaces are one-dimensional and align with the canonical basis of the latent space – a powerful yet frequently challenging or unattainable condition in real-world scenarios – thus narrowing the applicability of disentanglement. Addressing this, we propose a novel evaluation scheme, Importance-Weighted Orthogonality (IWO), to gauge the mutual orthogonality between subspaces encoding the data generating factors, irrespective of their dimensionality or alignment with the canonical basis. For that matter, we introduce a new method, Latent Orthogonal Analysis (LOA), which identifies the subspace encoding each data generating factor and establishes an importance-ranked basis spanning it, thereby forming the foundational bedrock for IWO. Through extensive comparisons of learned representations from synthetic and real-world datasets, we demonstrate that, relative to existing disentanglement metrics, IWO offers a superior assessment of orthogonality and exhibits stronger correlation with downstream task performance across a spectrum of applications.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 9436
Loading