Human-Prior Correction: Scalable Post-hoc Calibration that Aligns Vision Models with Human Uncertainty

Anitej Thamma, Shreyas Krishnan

Keywords: uncertainty calibration, post-hoc calibration, human uncertainty, human priors, log-linear pooling, foundation model priors (CLIP/DINO), conformal prediction, distribution shift robustness, subgroup fairness, vision classification

TL;DR: HPC aligns model confidence with human uncertainty by log-linear pooling of model probabilities with human or CLIP-derived confusion priors, improving calibration, robustness under shift, and conformal set tightness without retraining.

Abstract: Deep vision models achieve high accuracy but produce poorly-calibrated predictions that misalign with human uncertainty, limiting their reliability in safety-critical applications. We propose Human-Prior Correction (HPC), a post-hoc calibration method that aligns model confidence with human perceptual uncertainty without retraining. HPC solves a principled Bayesian objective $\min_p \text{KL}(p||\text{human}) + \lambda \text{KL}(p||\text{model})$ yielding the closed-form solution $p^* \propto \text{human}^\alpha \cdot \text{model}^{1-\alpha}$, where the human confusion prior captures systematic perceptual similarities (e.g., cat$\leftrightarrow$dog). Our key insight is that foundation models like CLIP encode human-like confusion patterns that serve as proxy priors, eliminating the need for expensive human annotations. Across CIFAR-10/100 and ImageNet, HPC achieves (1) 19.7\% improvement in human alignment (NLL$_\text{human}$), (2) 23.9\% ECE reduction while maintaining accuracy, (3) increasing benefits under distribution shift (21\% improvement at maximum corruption), and (4) 45\% reduction in worst-group calibration disparity. The method adds negligible computational overhead (less than 0.1\% of forward pass), combines synergistically with existing calibration techniques, and improves conformal prediction sets by yielding tighter intervals at fixed coverage. By incorporating structured human confusions into predictions, HPC bridges the gap between statistical calibration and human-aligned uncertainty, a critical step toward trustworthy AI deployment.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 23068