Conformal Prediction Sets for Reliable Uncertainty Quantification in Natural Language Processing

Rui Luo

Abstract: Natural language processing (NLP) classification tasks often benefit from predicting a set of possible labels with confidence scores to capture uncertainty. However, existing methods struggle with the high-dimensional and sparse nature of textual data. We propose a novel conformal prediction method designed for NLP that utilizes confidence scores from deep learning models to construct prediction sets. Our approach achieves the coverage rate while managing the size of the prediction sets. Through theoretical analysis and extensive experiments, we demonstrate that our method outperforms existing techniques on various datasets, providing reliable uncertainty quantification for NLP classifiers. We contribute a novel conformal prediction method, theoretical analysis, and empirical evaluation. Our work advances the practical deployment of NLP systems by enabling reliable uncertainty quantification.

Submission Number: 133