An Empirical Investigation of Contextualized Number Prediction

Daniel Spokoyny, Taylor Berg-Kirkpatrick

Abstract:

We conduct a large scale empirical investigation of contextualized number prediction in
running text. Specifically, we consider two
tasks: (1) masked number prediction – predicting a missing numerical value within a sentence, and (2) numerical anomaly detection –
detecting an errorful numeric value within a
sentence. We experiment with novel combinations of contextual encoders and output distributions over the real number line. Specifically, we introduce a suite of output distribution parameterizations that incorporate latent
variables to add expressivity and better fit the
natural distribution of numeric values in running text, and combine them with both recurrent and transformer-based encoder architectures. We evaluate these models on two numeric datasets in the financial and scientific
domain. Our findings show that output distributions that incorporate discrete latent variables and allow for multiple modes outperform simple flow-based counterparts on all
datasets, yielding more accurate numerical prediction and anomaly detection. We also show
that our models effectively utilize textual context and benefit from general-purpose unsupervised pretraining