Model Diffusion for Certifiable Few-shot Transfer Learning

ICLR 2026 Conference Submission18103 Authors

Keywords: trustworthy AI, safety, theoretical guarantees, risk certificates, pac-bayes, LLM, VLM, few-shot learning

TL;DR: Provable risk certificates for LLMs and VLMs cross-task low-shot transfer learning by sampling and evaluating a generative diffusion model trained over training tasks peft adapters

Abstract: In contemporary deep learning, a prevalent and effective workflow for solving low-data problems is adapting powerful pre-trained foundation models (FMs) to new tasks via parameter-efficient fine-tuning (PEFT). However, while empirically effective, the resulting solutions lack generalisation guarantees to certify their accuracy - which may be required for ethical or legal reasons prior to deployment in high-importance applications. In this paper we develop a novel transfer learning approach that is designed to facilitate non-vacuous learning theoretic generalisation guarantees for downstream tasks, even in the low-shot regime. Specifically, we first use upstream tasks to train a {\em distribution over PEFT parameters}. We then learn the downstream task by a {\em sample-and-evaluate} procedure -- sampling plausible PEFTs from the trained diffusion model and selecting the one with the highest likelihood on the downstream data. Crucially, this confines our model hypothesis to a {\em finite} set of PEFT samples. In contrast to the typical continuous hypothesis spaces of neural network weights, this facilitates tighter risk certificates. We instantiate our bound and show non-trivial generalization guarantees compared to existing learning approaches which lead to vacuous bounds in the low-shot regime.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 18103