RECAL: Sample-Relation Guided Confidence Calibration over Tabular Data

Wang HaoTian, Zhen Zhang, Mengting Hu, Qichao Wang, Liang Chen, Yatao Bian, Bingzhe Wu

Submission Type: Regular Long Paper

Submission Track: Machine Learning for NLP

Submission Track 2: Interpretability, Interactivity, and Analysis of Models for NLP

Keywords: Confidence Calibration, Tabular Data, Element-Wise Temperature Scaling

TL;DR: A general post-training framework, RECAL, calibrates the confidence of ML models. It models relations between table samples, using an element-wise scaling approach to improve confidence.

Abstract: Tabular-format data is widely adopted in various real-world applications. Various machine learning models have achieved remarkable success in both industrial applications and data-science competitions. Despite these successes, most current machine learning methods for tabular data lack accurate confidence estimation, which is needed by some high-risk sensitive applications such as credit modeling and financial fraud detection. In this paper, we study the confidence estimation of machine learning models applied to tabular data. The key finding of our paper is that a real-world tabular dataset typically contains implicit sample relations, and this can further help to obtain a more accurate estimation. To this end, we introduce a general post-training confidence calibration framework named RECAL to calibrate the predictive confidence of current machine learning models by employing graph neural networks to model the relations between different samples. We perform extensive experiments on tabular datasets with both implicit and explicit graph structures and show that RECAL can significantly improve the calibration quality compared to the conventional method without considering the sample relations.

Submission Number: 789