Embedding game: dimensionality reduction as a two-person zero-sum game

Kyle Luther

Keywords: manifold learning, game theory, embedding, nonlinear dimensionality reduction

Abstract: Dimensionality reduction is often formulated as a minimization containing a sparse sum of attractive interactions and a dense sum of repulsive interactions $\sum_{ij} f(\Vert \mathbf{y}_i - \mathbf{y}_j \Vert)$ between embedding vectors. This dense sum is usually subsampled to avoid computing all $N^2$ terms. In this paper we provide a novel approximation to the repulsive sum by deriving a landmark-based lower bound and then maximizing this lower bound with respect to the landmarks. After inserting this approximation into the original objective we are left with a minimax problem where the embedding vectors minimize the objective by pulling on their neighbors and running away from the landmarks while the landmarks maximize the objective by pulling on the embedding vectors and running away from other nearby landmarks. We use gradient descent ascent to find saddle points and show that our method can produce high quality visualizations without ever explicitly computing any pairwise repulsion between embedding vectors.

TL;DR: We formulate a class of nonlinear embedding problems as a game between embedding vectors and landmark vectors