Predictive Distribution Estimation for Bayesian Machine Learning using a Dirichlet Process Prior

Paul Rademacher, Milos Doroslovacki

Abstract: In Bayesian treatments of machine learning, the success or failure of the estimator/classifier hinges on how well the prior distribution selected by the designer matches the actual data-generating model. This paper assumes that the model distribution is a realization of a Dirichlet process and assesses the mismatch between the true predictive distribution and the predictive distribution approximated using the training data. It is shown that highly localized Dirichlet priors can overcome the burden of a limited training set when the prior mean is well matched to the true distribution, but will degrade the approximation if the match is poor. A bias/variance trade-off will be demonstrated with illustrative examples.