Sparse Activations as Conformal Predictors

Margarida M Campos, João Cálem, Sophia Sklaviadis, Mario A. T. Figueiredo, Andre Martins

TL;DR: Temperature scaling of sparse activations as conformal prediction: finding new conformity scores and motivating new threads of research for uncertainty quantification.

Abstract:

Conformal prediction is a distribution-free framework for uncertainty quantification that replaces point predictions with sets, offering marginal coverage guarantees (i.e., ensuring that the sets contain the true label with a specified probability, in expectation).
In this paper, we uncover a novel connection between conformal prediction and sparse "softmax-like" transformations, such as sparsemax and $\gamma$-entmax (with $\gamma> 1$), which assign nonzero probability only to some labels.
We introduce new non-conformity scores for classification which make the calibration process correspond to the widely used temperature scaling method. At test time, applying these sparse transformations with the calibrated temperature leads to a support set (i.e., the set of labels with nonzero probability) that automatically inherits the coverage guarantees of conformal prediction.
Through experiments on computer vision and text classification benchmarks, we demonstrate that the proposed method achieves competitive results in terms of coverage, efficiency, and adaptiveness compared to standard non-conformity scores based on softmax.

Submission Number: 901