Conformity Score Averaging for Classification

Rui Luo, Zhixin Zhou

TL;DR: A model averaging method that uses subclass of functions in Vapnik-Chervonenkis theory that achieves state-of-the-art prediction set efficiency in conformal prediction.

Abstract: Conformal prediction provides a robust framework for generating prediction sets with finite-sample coverage guarantees, independent of the underlying data distribution. However, existing methods typically rely on a single conformity score function, which can limit the efficiency and informativeness of the prediction sets. In this paper, we present a novel approach that enhances conformal prediction for multi-class classification by optimally averaging multiple conformity score functions. Our method involves assigning weights to different score functions and employing various data splitting strategies. Additionally, our approach bridges concepts from conformal prediction and model averaging, offering a more flexible and efficient tool for uncertainty quantification in classification tasks. We provide a comprehensive theoretical analysis grounded in Vapnik–Chervonenkis (VC) theory, establishing finite-sample coverage guarantees and demonstrating the efficiency of our method. Empirical evaluations on benchmark datasets show that our weighted averaging approach consistently outperforms single-score methods by producing smaller prediction sets without sacrificing coverage.

Lay Summary: We present a novel approach for improving conformal prediction in multi-class classification by optimally combining multiple uncertainty scoring functions rather than relying on a single one. Conformal prediction generates prediction sets with guaranteed coverage probability, but existing single-score methods can produce inefficiently large sets. Our method assigns optimal weights to different scoring functions using four data splitting strategies (VFCP, EFCP, DLCP, DLCP+) that balance coverage guarantees with prediction set efficiency. We provide theoretical analysis using Vapnik-Chervonenkis theory establishing finite-sample coverage guarantees and demonstrating near-optimal efficiency as dataset size increases. Experiments on benchmark datasets (CIFAR-10/100, MNIST, Fashion-MNIST, ImageNet) show our weighted averaging approach consistently produces smaller prediction sets than single-score baselines while maintaining required coverage, making it particularly valuable for applications requiring precise uncertainty quantification with more informative predictions.

Primary Area: General Machine Learning

Keywords: conformal prediction, model averaging, VC dimension

Submission Number: 9642