Obtaining calibrated probability using ROC Binning

Meesun Sun, Sungzoon Cho

Abstract: Obtaining calibrated probability, or actual occurrence, is crucial in many real problems because it effectively supports the decision-making process with good assessment of cost and effect. Estimating calibrated probability is a more significant issue in class imbalance and class overlap problems, where direct application of classification algorithms may result in substantial errors. Consequently, several post-processing calibration techniques that aim at improving the probability estimation or the error distribution of existing classification models have been developed. In this underlying context, we propose Receiver Operating Characteristics Binning, a robust method that provides accurate calibrated probabilities that are robust to changes in the prevalence of the positive class by using a combination of True Positive Rate, False Positive Rate, and the prevalence of the positive class. The results of experiments conducted on the real-world UCI dataset indicate that, given a training set in which the positive class proportion is noticeably different from that of the test set, the proposed ROC Binning method outperforms conventional calibration methods.