Ordered Stick-Breaking Prior for Sequential MCMC Inference of Bayesian Nonparametric Models

Mrinal Das, Trapit Bansal, Chiranjib Bhattacharyya

Abstract:

This paper introduces ordered stick-breaking
process (OSBP), where the atoms in a stick
breaking process (SBP) appear in order. The
choice of weights on the atoms of OSBP en
sure that; (1) probability of adding new atoms
exponentially decrease, and (2) OSBP, though
non-exchangeable, admit predictive probability
functions (PPFs). In a Bayesian nonparamet
ric (BNP) setting, OSBP serves as a natural
prior over sequential mini-batches, facilitating
exchange of relevant statistical information by
sharing the atoms of OSBP. One of the major
contributions of this paper is SUMO, an MCMC
algorithm, for solving the inference problem
arising from applying OSBP to BNP models.
SUMOusesthePPFsofOSBPtoobtainaGibbs
sampling based truncation-free algorithm which
applies generally to BNP models. For large scale
inference problems existing algorithms such as
particle filtering (PF) are not practical and vari
ational procedures such as TSVI (Wang & Blei,
2012) are the only alternative. For Dirichlet pro
cess mixture model (DPMM), SUMO outper
forms TSVI on perplexity by 33% on 3 datasets
with million data points, which are beyond the
scope of PF, using only 3GB RAM.