Learning single index models via harmonic decomposition

Nirmit Joshi, Hugo Koubbi, Theodor Misiakiewicz, Nathan Srebro

Keywords: single-index models, statistical and computational complexity

TL;DR: We revisit the single-index models and argue that spherical harmonics, not Hermite polynomials, are a natural basis. We characterize the complexity for any spherically symmetric input measure, & provide several new insights for the Gaussian case.

Abstract: We study the problem of learning single-index models, where the label $y \in \mathbb{R}$ depends on the input $\boldsymbol{x} \in \mathbb{R}^d$ only through an unknown one-dimensional projection $\langle \boldsymbol{w_*}, \boldsymbol{x} \rangle$. Prior work has shown that under Gaussian inputs, the statistical and computational complexity of recovering $\boldsymbol{w}_*$ is governed by the Hermite expansion of the link function. In this paper, we propose a new perspective: we argue that *spherical harmonics*---rather than *Hermite polynomials*---provide the natural basis for this problem, as they capture its intrinsic \textit{rotational symmetry}. Building on this insight, we characterize the complexity of learning single-index models under arbitrary spherically symmetric input distributions. We introduce two families of estimators---based on tensor-unfolding and online SGD---that respectively achieve either optimal sample complexity or optimal runtime, and argue that estimators achieving both may not exist in general. When specialized to Gaussian inputs, our theory not only recovers and clarifies existing results but also reveals new phenomena that had previously been overlooked.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 6957