Neural Manifold Clustering and Embedding

ZENGYI LI, Yubei Chen, Yann LeCun, Friedrich Sommer

Keywords: Self-supervised Learning, Clustering, Subspace Clustering, Manifold Learning, Deep Subspace Clustering

Abstract: Given a union of non-linear manifolds, non-linear subspace clustering or manifold clustering aims to cluster data points based on manifold structures and also learn to parameterize each manifold as a linear subspace in a feature space. Deep neural networks have the potential to achieve this goal under highly non-linear settings given their large capacity and flexibility. We argue that achieving manifold clustering with neural networks requires two essential ingredients: a domain-specific constraint that ensures the identification of the manifolds, and a learning algorithm for embedding each manifold to a linear subspace in the feature space. This work shows that many constraints can be implemented by data augmentation. For subspace feature learning, Maximum Coding Rate Reduction (MCR$^2$) objective can be used. Putting them together yields Neural Manifold Clustering and Embedding (NMCE), a novel method for general purpose manifold clustering, which significantly outperforms autoencoder-based deep subspace clustering and achieve state-of-the-art performance on several important benchmarks. Further, on more challenging natural image datasets, NMCE can also outperform other algorithms specifically designed for clustering. Qualitatively, we demonstrate that NMCE learns a meaningful and interpretable feature space. As the formulation of NMCE is closely related to several important Self-supervised learning (SSL) methods, we believe this work can help us build a deep understanding on SSL representation learning.

One-sentence Summary: We show that to achieve general manifold clustering and embedding with neural networks, one need to combine a constraint implemented by data augmentation with a subspace feature learning method.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2201.10000/code)