Multitask Learning with Stochastic Interpolants

Hugo Negrel, Florentin Coeurdoux, Michael Samuel Albergo, Eric Vanden-Eijnden

Keywords: generative models, flows, diffusions, conditional generation, pinpointing, fine-tuning, multiscale generation, zero-shot, multitask

TL;DR: We introduce operator-based interpolants—a novel framework that replaces time variables with linear operators in generative models—enabling a single model to perform multiple tasks without task-specific training.

Abstract: We propose a framework for learning maps between probability distributions that broadly generalizes the time dynamics of flow and diffusion models. To enable this, we generalize stochastic interpolants by replacing the scalar time variable with vectors, matrices, or linear operators, allowing us to bridge probability distributions across multiple dimensional spaces. This approach enables the construction of versatile generative models capable of fulfilling multiple tasks without task-specific training. Our operator-based interpolants not only provide a unifying theoretical perspective for existing generative models but also extend their capabilities. Through numerical experiments, we demonstrate the zero-shot efficacy of our method on conditional generation and inpainting, fine-tuning and posterior sampling, and multiscale modeling, suggesting its potential as a generic task-agnostic alternative to specialized models.

Primary Area: Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Submission Number: 26950