Minimizing Dependence between Embedding Dimensions with Adversarial Networks

Pierre-François De Plaen, Marc Proesmans, Tinne Tuytelaars, Luc Van Gool

Keywords: representation learning, adversarial networks, independence, information maximization, generalization

Abstract: Learning representations with minimally dependent embedding dimensions can have many potential benefits such as improved generalization and interpretability. This work provides a differentiable and scalable algorithm for dependence minimization, moving beyond existing linear pairwise decorrelation methods. Our algorithm involves an adversarial game where small networks identify dimension relationships, while the main model exploits this information to reduce dependencies. We empirically verify that the algorithm converges. We then explore dependence reduction as a proxy for maximizing information content. We showcase the algorithm's effectiveness on the Clevr-4 dataset, both with and without supervision, and achieve promising results on the ImageNet dataset. Finally, we propose an algorithm modification that gives more control over the level of dependency, sparking a discussion on optimal redundancy levels for specific applications. Although the algorithm performs well on synthetic data, further research is needed to optimize it for tasks such as out-of-distribution detection.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7756