Conformal Prediction for Text Infilling and Part-of-Speech Prediction

Max Lovig, Neil Dey, Carolina Kapper, Jing Ding, Jack Ferrel, Emi Planchon, Jonathan P. Williams

Abstract:

Modern machine learning algorithms are capable of providing remarkably accurate point-predictions; however, questions
remain about their statistical reliability. Unlike conventional machine learning methods, conformal prediction algorithms
return confidence sets (i.e., set-valued predictions) that correspond to a given significance level. Moreover, these confidence
sets are valid in the sense that they guarantee finite sample control over type 1 error probabilities, allowing the practitioner
to choose an acceptable error rate. In our paper, we propose inductive conformal prediction (ICP) algorithms for the tasks
of text infilling and part-of-speech (POS) prediction for natural language data. We construct new ICP-enhanced algorithms
for POS tagging based on BERT (bidirectional encoder representations from transformers) and BiLSTM (bidirectional
long short-term memory) models. For text infilling, we design a new ICP-enhanced BERT algorithm. We analyze the
performance of the algorithms in simulations using the Brown Corpus, which contains over 57,000 sentences. Our results
demonstrate that the ICP algorithms are able to produce valid set-valued predictions that are small enough to be applicable
in real-world applications. We also provide a real data example for how our proposed set-valued predictions can improve
machine generated audio transcriptions.