Wasserstein Gradient Boosting: A Framework for Distribution-Valued Supervised Learning

Takuo Matsubara

Keywords: Gradient Boosting; Wasserstein Gradient Flow; Uncertainty Quantification

TL;DR: We propose a novel family of gradient boosting, Wasserstein gradient boosting, which returns a set of particles that approximates a target probability distribution assigned at each input.

Abstract: Gradient boosting is a sequential ensemble method that fits a new weaker learner to pseudo residuals at each iteration. We propose Wasserstein gradient boosting, a novel extension of gradient boosting, which fits a new weak learner to alternative pseudo residuals that are Wasserstein gradients of loss functionals of probability distributions assigned at each input. It solves distribution-valued supervised learning, where the output values of the training dataset are probability distributions. In classification and regression, a model typically returns, for each input, a point estimate of a parameter of a noise distribution specified for a response variable, such as the class probability parameter of a categorical distribution specified for a response label. A main application of Wasserstein gradient boosting in this paper is tree-based evidential learning, which returns a distributional estimate of the response parameter for each input. We empirically demonstrate the competitive performance of the probabilistic prediction by Wasserstein gradient boosting in comparison with existing uncertainty quantification methods.

Primary Area: Probabilistic methods (for example: variational inference, Gaussian processes)

Submission Number: 20807