Are you using test log-likelihood correctly?

Sameer Deshpande, Soumya Ghosh, Tin D. Nguyen, Tamara Broderick

Keywords: Test log-likelihood, predictive mean, posterior approximation quality

TL;DR: We present simple examples demonstrating how comparisons based on test log-likelihood can contradict comparisons according to other objectives

Abstract: Test log-likelihood is commonly used to compare different models of the same data and different approximate inference algorithms for fitting the same probabilistic model. We present simple examples demonstrating how comparisons based on test log-likelihood can contradict comparisons according to other objectives. Specifically, our examples show that (i) conclusions about forecast accuracy based on test log-likelihood comparisons may not agree with conclusions based on other distributional quantities like means; and (ii) that approximate Bayesian inference algorithms that attain higher test log-likelihoods need not also yield more accurate posterior approximations.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/are-you-using-test-log-likelihood-correctly/code)