Continuously Parameterized Mixture Models

Christopher M Bender, Yifeng Shi, Marc Niethammer, Junier Oliva

Keywords: mixture models, normalizing flows, ordinary differential equations, clustering, interpretable learning

TL;DR: We parameterize mixtures of factor analyzers by a neural ordinary differential equation and train with a smooth curriculum to learn an interpretable likelihood model superior to standard mixture results.

Abstract:

Mixture models are universal approximators of smooth densities but are difficult to utilize in complicated datasets due to restrictions on typically available modes and challenges with initialiations.
We show that by continuously parameterizing a mixture of factor analyzers using a learned ordinary differential equation, we can improve the fit of mixture models over direct methods.
Once trained, the mixture components can be extracted and the neural ODE can be discarded, leaving us with an effective, but low-resource model.
We additionally explore the use of a training curriculum from an easy-to-model latent space extracted from a normalizing flow to the more complex input space and show that the smooth curriculum helps to stabilize and improve results with and without the continuous parameterization.
Finally, we introduce a hierarchical version of the model to enable more flexible, robust classification and clustering, and show substantial improvements against traditional parameterizations of GMMs.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Generative models