Robust Transfer for Bayesian Optimization with Multi-Task Prior-Fitted Networks

ICLR 2026 Conference Submission20784 Authors

Keywords: Bayesian optimization, transfer learning

Abstract:

Bayesian optimization (BO) is a sample-efficient optimization technique for black-box optimization, and using transfer learning to leverage historical information from related tasks can greatly improve its performance. Multi-task Gaussian processes are commonly used to transfer knowledge from source tasks to target tasks, but these models often make strong assumptions about the relationships between tasks and thus suffer from negative transfer and degraded predictive performance when these assumptions are violated. In this paper, we present Multi-Task Prior-Data Fitted Networks (MTPFNs), a flexible surrogate model that emulates Bayesian inference over user-specified priors over the relationship between tasks. We also propose a novel data-generation procedure specifically designed for the Bayesian optimization transfer setting which enables MTPFNs to be robust to negative transfer and efficiently leverage relevant information.
Across a variety of synthetic and real-world benchmarks including hyperparameter optimization, we demonstrate that MTPFNs successfully transfer knowledge in challenging scenarios where existing multi-task Gaussian processes struggle, outperforming existing robust transfer learning methods for Bayesian optimization.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 20784