Mixtures of Conditional Maximum Entropy Models

Dmitry Pavlov, Alexandrin Popescul, David M. Pennock, Lyle H. Ungar

Abstract: Driven by successes in several application areas, maximumen tropy modeling has recently gained considerable popularity. We generalize the standard maximum entropy formulation of classification problems to better handle the case where complex data distributions arise from a mixture of simpler underlying (latent) distributions. We develop a theoretical framework for characterizing data as a mixture of maximum entropy models. We formulate a maximum-likelihood interpretation of the mixture model learning, and derive a generalized EM algorithm to solve the corresponding optimization problem. We present empirical results for a number of data sets showing that modeling the data as a mixture of latent maximumen tropy models gives significant improvement over the standard, single component, maximum entropy approach.