Subspace Clustering via Robust Self-Supervised Convolutional Neural Network

Dario Sitnik, Ivica Kopriva

Keywords: deep subspace clustering, convolutional neural networks, self-supervised learning, correntropy, generalization, robustness

Abstract:

Subspace clustering (SC) approaches based on the self-representation model achieved encouraging performance when compared with the clustering algorithms that rely on proximity measures between data points. However, they still face serious limitations in real-world applications. One limitation relates to the linearity assumption of the self-representation model. The reason is that, usually, samples lie in non-linear manifolds, e.g. face images acquired under non-uniform illumination and different poses. Another limitation relates to errors that can be random or sample-specific (outliers), whereas in real-world applications their origin is mostly unknown. Furthermore, the majority of existing algorithms use external clustering validation methods to measure clustering quality, and that requires access to ground-truth (GT) labels. Hence, it is practically important to develop a deep SC method that jointly learns self-expressive (SE) feature representation and handles data corruptions of unknown origin, and estimates clustering error using the internal clustering validation method, i.e. without access to the GT. Mostly, the base of the recently developed deep SC networks is convolutional autoencoder. It is an end-to-end fully convolutional network that is based on the minimization of the reconstruction error. Together, the autoencoder and an additional SE module are forming a Deep SC network (DSCNet). Hence, the total loss function of DSCNet is composed of reconstruction loss and SE loss. That is, during the learning process, the quality of clustering is not taken into account. Self-supervised convolutional SC network ($S^2$ConvSCN) addressed this issue through the addition of a fully connected layer (FC) module and a spectral clustering module that, respectively, generate soft- and pseudo-labels.

While inheriting the architecture of the $S^2$ConvSCN, this paper proposes a robust SE loss and an early self-stopping criterion for training. Robustness to arbitrary (unknown) types of data corruptions is achieved by using the correntropy induced metric (CIM) of the error of the SE model. By mapping the input data space to a reproducible kernel Hilbert space (RKHS), correntropy defines an $\ell_2$ distance in RKHS and creates nonlinear distance measure in the original input data space. Hence, correntropy is the optimal measure for error with the arbitrary non-Gaussian distribution as opposed to the MSE that implies Gaussian distribution of errors. Thus, because it is estimated directly from data, CIM can handle data corruption of unknown origin. As opposed to the DSCNet and $S^2$ConvSCN, the self-stopping criterion of the proposed algorithm is achieved by reaching a plateau in the change of loss value.

Although the architecture of $S^2$ConvSCN contains the FC module, which is capable of handling out-of-sample data by using a softmax classifier, its performance has not been tested on unseen data. The importance of the FC module is, however, emphasized through self-supervision. In a truly unsupervised setting, pseudo-labels generated from the spectral clustering module guide the learning process through the induced classification error at the end of the FC module. They also enhance the goodness of the self-representation matrix through penalizing incorrectly connected elements. Adding these two loss functions to the total loss makes pseudo-labels an especially important component in label-constrained applications. Such dual self-supervision, in theory, enables deep networks to learn representation from available unlabeled data and to use the small number of labeled data to validate the trained model.

In addition to the main contributions of the paper, this study has three side-contributions. It aimed to set up a more transparent way for the proper performance estimation of deep SC models and to integrate block-diagonal regularization into the gradient descent learning process. The majority of SC studies optimize hyperparameters using external clustering validation methodology, whereas the same labels are used for tuning hyperparameters and evaluating the final clustering performance. Furthermore, models like $S^2$ConvSCN suffer from an early commitment problem as they depend on weight-transfer from pretrained models that have “seen” the GT already (e.g. DSCNet transferred to $S^2$ConvSCN). From the machine learning perspective, data leakage and hyperparameter tuning based on the test GT are unacceptable. Measured clustering performances of $S^2$ConvSCN and DSCNet on the out-of-sample (unseen) data, thus, differ significantly from the optimistic one presented in the original papers. Also, post-processing of self-representation matrix is reduced to a significant extent. Robust $S^2$ConvSCN outperforms its baseline version by a significant amount for both seen and unseen data on four well-known datasets. To the best of our knowledge, such an ablation study on unseen data has not been reported previously in SC studies.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: This paper proposes a robust formulation of the self-supervised convolutional subspace clustering network, with enhanced generalization capability, by using the correntropy induced metric of the error and early-stopping technique.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=_sbSwE8Cwk