On the Uncertainty in Prior-Data Fitted Network Pretraining

Manuel Hülskamp, Julius Kobialka, Emanuel Sommer, David Rügamer

Keywords: Prior-Data Fitted Networks, Bayesian Deep Ensembles, Tabular Data, Weight-Space Uncertainty, MCMC

TL;DR: We study uncertainty in Prior-data fitted networks, uncovering two unaddressed sources of uncertainty, and propose methods to address uncertainties arising in the pretraining process.

Abstract: Prior-data fitted networks (PFNs) have recently emerged as a new paradigm for supervised machine learning by approximating Bayesian posterior predictive distributions via in-context learning. While leading to state-of-the-art predictive performance across a wide range of benchmarks, quantifying the approach's uncertainty is challenging. Using an empirical risk minimization perspective, we characterize possible sources of uncertainty in PFN predictions and empirically show that certain uncertainties cannot be reduced by simply scaling the model size or pretraining. To explicitly address these overlooked sources of uncertainty, we study deep ensembles of PFNs and formulate a Bayesian neural network version of PFNs for which we obtain samples from an approximate posterior via Markov chain Monte Carlo.

Submission Number: 73