On the Uncomputability of Partition Functions in Energy-Based Sequence Models

Chu-Cheng Lin, Arya D. McCarthy

Keywords: energy-based models, turing completeness, model capacity, sequence models, autoregressive models, partition function, parameter estimation, model selection

Abstract: In this paper, we argue that energy-based sequence models backed by expressive parametric families can result in uncomputable and inapproximable partition functions. Among other things, this makes model selection--and therefore learning model parameters--not only difficult, but generally _undecidable_. The reason is that there are no good deterministic or randomized estimates of partition functions. Specifically, we exhibit a pathological example where under common assumptions, _no_ useful importance sampling estimates of the partition function can guarantee to have variance bounded below a rational number. As alternatives, we consider sequence model families whose partition functions are computable (if they exist), but at the cost of reduced expressiveness. Our theoretical results suggest that statistical procedures with asymptotic guarantees and sheer (but finite) amounts of compute are not the only things that make sequence modeling work; computability concerns must not be neglected as we consider more expressive model parametrizations.

One-sentence Summary: EBMs over sequences have several theoretical limitations as learnable probabilistic sequence models.