Decentralized efficient nonparametric stochastic optimization

Alec Koppel, Santiago Paternain, Cédric Richard, Alejandro Ribeiro

Abstract: We consider stochastic optimization problems defined over reproducing kernel Hilbert spaces (RKHS), where a multi-agent network aims to learn decision functions, i.e., nonlinear statistical models, that are optimal in terms of a global convex functional that aggregates data across the network, while only having access to locally observed sequentially available training examples. We address this problem by allowing each agent to learn a local regression function while enforcing consensus constraints. We use a penalized variant of functional stochastic gradient descent operating simultaneously with low-dimensional subspace projections. The resulting algorithm allows each individual agent to learn, based upon its locally observed data stream and message passing with its neighbors, a function that is provably close to globally optimal and satisfies the consensus constraints. Moreover, when constant learning rates are used, the complexity of the learned regression functions is guaranteed to be finite. For a multi-class kernel logistic regression task with Gaussian mixtures data, we observe stable function estimation and state of the art accuracy for distributed online multi-class classification.