REDISTRIBUTING FOUNDATION MODEL LOGITS FOR BETTER DOMAIN GENERALIZATION IN LOW-SHOT CLASSIFICATION

Behraj Khan, Behroz Mirza, Tahir Syed

Keywords: foundation models, calibration, distribution shift

Abstract: Confidence calibration is an emerging challenge in real-world decision systems that repurpose foundations models for downstream vision classification tasks. Due to various reasons ..., logit scores on the CLIP head remain large irrespective of whether the image-language pairs reconcile. Ideally, they should be proportional to that reconciliation. This paper adaptively regulates that 'temperature'. We propose a penalty incorporated into loss objective that penalizes incorrect classifications whenever one is made during finetuning, by moving an amount of log-likelihood to the true class commensurate to the relative amplitudes of the two likelihoods. We refer to it as \textit{confidence misalignment penalty (CMP)}. Extensive experiments on $12$ vision datasets and $5$ domain generalization datasets supports the calibration performance of our method against stat-of-the-art. CMP outperforms the benchmarked prompt learning methods, demonstrating average improvement in Expected Calibration Error (ECE) by average $6.01$\%, $4.01$ \% at minimum and $9.72$\% at maximum.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 13472