Dimension reduction as an optimization problem over a set of generalized functions

Rustem Takhanov

Keywords: Unsupervised dimension reduction, sufficient dimension reduction, alternating scheme, Fourier transform, maximum mean discrepancy, Wasserstein distance, positive definite functions, Bochner’s theorem

Abstract:

We reformulate unsupervised dimension reduction problem (UDR) in the language of tempered distributions, i.e. as a problem of approximating an empirical probability density function $p_{\rm{emp}}({\mathbf x})$ by another tempered distribution $q({\mathbf x})$ whose support is in a $k$-dimensional subspace. Thus, our problem is reduced to the minimization of the distance between $q$ and $p_{\rm{emp}}$, $D(q, p_{\rm{emp}})$, over a pertinent set of generalized functions.

This infinite-dimensional formulation allows to establish a connection with another classical problem of data science --- the sufficient dimension reduction problem (SDR). Thus, an algorithm for the first problem induces an algorithm for the second and vice versa. In order to reduce an optimization problem over distributions to an optimization problem over ordinary functions we introduce a nonnegative penalty function $R(f)$ that ``forces'' the support of $f$ to be $k$-dimensional.
Then we present an algorithm for minimization of $I(f)+\lambda R(f)$, based on the idea of two-step iterative computation, briefly described as a) an adaptation to real data and to fake data sampled around a $k$-dimensional subspace found at a previous iteration, b) calculation of a new $k$-dimensional subspace. We demonstrate the method on 4 examples (3 UDR and 1 SDR) using synthetic data and standard datasets.

One-sentence Summary: We reformulate unsupervised dimension reduction problem and sufficient dimension reduction problem in the language of tempered distributions and present a new optimization method.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=GcbGa6NIL4