Out-of-Sample Extension of Spectral Embeddings: An Optimization Perspective

Chester Holtz, Taanish Reja, Gal Mishne

Keywords: Manifold learning, Riemannian optimization

TL;DR: We propose a new framework for out-of-sample extensions for spectral graph-based embedding algorithms motivated from the perspective of the underlying eigenvector problem as a quadratic program on the Stiefel Manifold.

Abstract:

Graph-based manifold learning constructs / reveals low dimensional embeddings of high-dimensional data, however requires out-of-sample-extension methods to embed new data points.
We propose a new framework, ROSE (Riemannian Out-of-Sample Extension), for out-of-sample extensions for spectral graph-based embedding algorithms. ROSE is motivated from an optimization perspective of the underlying eigenvector problem associated with classic manifold learning problems. Similar to graph-based semi-supervised learning, our approach exploits the geometry of the unlabeled samples in addition to the labeled samples, by treating the in-sample embeddings as \emph{labeled} data. Despite its nonconvexity, ROSE is solvable by first-order methods, which converge to global minimizers under certain assumptions. We present numerical experiments on a variety of real-world and synthetic benchmarks. Empirically, we show on standard image datasets (MNIST, FMNIST, CIFAR-10) that ROSE closely approximates the ground-truth eigenvectors in low and high label rate regimes with respect to neighborhood preservation.

Submission Type: Extended abstract (max 4 main pages).

Submission Number: 155