When Aggregation Fails: From PAC-Bayes Theory to Practical Selection for Conformal Prediction

Pauline Bourigault, Danilo Mandic

Keywords: conformal prediction, PAC-Bayes

Abstract: We identify and characterize a fundamental incompatibility between PAC-Bayes theory and conformal prediction: while PAC-Bayes minimizes average risk through posterior aggregation, conformal prediction's efficiency depends on quantile behavior. We prove that this \emph{average-quantile divergence} phenomenon causes standard PAC-Bayes aggregation to systematically select suboptimal models for conformal prediction, with linear aggregation methods unable to preserve quantile optimality and efficiency losses proportional to both posterior entropy and score heterogeneity. To address this limitation, we develop PAC-Bayes Informed Selection (PBIS), which uses quantile-aware posteriors for model selection rather than aggregation. We establish PAC-Bayes bounds for quantile functionals requiring novel techniques to handle their non-differentiable nature, and prove that PBIS achieves selection consistency with $O(\sqrt{T \log |\Theta|})$ regret in online settings. Empirical validation across 27 datasets demonstrates that PBIS achieves the narrowest prediction intervals among nine conformal methods while maintaining valid coverage, with 7.3\% average improvement in high-divergence scenarios versus 2.1\% in low-divergence ones compared to standard PAC-Bayes aggregation. The method maintains computational efficiency comparable to split conformal while being 82$\times$ faster than CQR. In online settings with distribution shifts, PBIS uniquely maintains valid coverage across gradual, sudden, and recurring shifts where competing adaptive methods fail. Our theoretical and empirical results establish that selection-based approaches fundamentally outperform aggregation for conformal prediction by avoiding the mathematical incompatibility between average risk and quantile optimization.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 9952