Roundoff Error in Metropolis-Hastings Accept-Reject Steps

Matthew Douglas Hoffman

Keywords: Metropolis-Hastings, Numerics, MCMC

TL;DR: Single-precision arithmetic can cause problems for Metropolis-Hastings algorithms, especially if you have lots of data.

Abstract: In this note, we consider what happens to the Metropolis-Hastings (M-H) algorithm when it is fed log-density calculations that are subject to roundoff error and catastrophic cancellations. We will briefly review the nature of these errors and how they can arise in M-H algorithms. Next, we will develop a theoretical model of roundoff error in M-H corrections, and find that it can lead to exponentially low acceptance rates in the magnitude of the errors (if the errors have Gaussian tails) and bias (if different types of states produce different types of errors). Finally, we will discuss some consequences of this phenomenon, and touch on some practical ways to avoid these consequences of catastrophic cancellations in software.