GAPA: Post-Hoc Uncertainty Quantification for Pre-Trained Models via Activation-Space Gaussian Processes

ICLR 2026 Conference Submission18955 Authors

Keywords: Uncertainty quantification, Gaussian processes, post-hoc methods, epistemic uncertainty, neural network activations, foundation models, calibration, out-of-distribution detection

TL;DR: We introduce GAPA, a novel method that adds single-pass, scalable uncertainty to frozen neural networks by modeling activations with Gaussian Processes.

Abstract: Weight-space uncertainty methods (BNNs, ensembles, Laplace) are difficult to apply post-hoc to frozen foundation models due to retraining requirements or prohibitive second-order computations. We introduce Gaussian Process Activations (GAPA), which replace deterministic activations with Gaussian processes whose prior mean equals the original nonlinearity—preserving predictions exactly while adding principled epistemic uncertainty. Using a 1-nearest-neighbour FITC surrogate with FAISS, GAPA yields closed-form, distance-aware uncertainties with $O(\log M)$ retrieval and $O(d)$ per-layer compute without sampling. Across regression, classification, segmentation, and language modelling (including GPT-2, 124M), GAPA matches or exceeds deep ensembles and Laplace in calibration and OOD detection while running 10–100× faster at test time. GAPA uniquely enables post-hoc deployment, single-pass inference, exact mean preservation, and scalable epistemic UQ for frozen backbones.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 18955