A Theoretical Insight of Histogram Binning and Extending to Multi-label Classification
Abstract: Learning well-calibrated probabilistic predictions is crucial as neural networks and machine learning models are increasingly employed in critical tasks nowadays. While there exist several post-processing methods aimed at calibrating output probabilities, most lack proper theoretical justification; in other words, they have typically only been validated on limited datasets and models to report empirical results. This work is divided into two parts. In the first part, we analyze some post-processing calibration methods from a geometrical perspective and demonstrate that calibrated outcomes consistently reduce Expected Calibration Error (ECE) while increasing accuracy. In the second part, we present a previously unexplored framework for calibrating the outcomes of multi-label problems by addressing multiple binary calibration problems. To achieve this, we introduce a novel concept of ECE for multi-label problems and provide substantial theoretical rationale for our approach. Experimental results demonstrate the feasibility and efficacy of our method in practice.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Yiming_Ying1
Submission Number: 2987
Loading