NTK-LoRA: Calibrating Fine-tuned Vision Transformers using Gaussian Processes

ICLR 2026 Conference Submission13185 Authors

Keywords: Calibration, Uncertainty estimation, Laplace approximation, Gaussian processes, Neural tangent kernel, Vision Transformers, LoRA fine-tuning

TL;DR: We introduce a simple post-hoc calibration method for fine-tuned Vision Transformers that leverages the Gaussian process view of the Laplace approximation.

Abstract:

Fine-tuning remains essential for adapting foundation models to domains where high precision is required, such as medical imaging or autonomous driving. However, this often leads to overconfident and poorly calibrated models, especially when fine-tuned on small datasets. We propose NTK-LoRA, a simple and effective post-hoc calibration method for fine-tuned Transformer models (e.g., Vision Transformers and LLMs) that leverages the Gaussian process view of neural networks to perform Laplace approximation of the posterior.
Our method is almost as straightforward to implement as temperature scaling (TS), requires no hyperparameter tuning or deeper expertise, allows incorporating prior knowledge through the choice of GP kernel, achieves better or comparable performance to TS and consistently outperforms Laplace calibration, which in our experiments often fails to improve over the baseline on binary classification.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 13185