Sampling from multimodal distributions with warm starts

ICLR 2026 Conference Submission19520 Authors

Keywords: Sampling, Simulated Tempering

TL;DR: We formalize the problem of sampling given warm starts and give the first proof for a polynomial-time algorithm in this setting.

Abstract:

Sampling from multimodal distributions is a central challenge in Bayesian inference and machine learning.
In light of hardness results for sampling---classical MCMC methods, even with tempering, can suffer from exponential mixing times---a natural question
is how to leverage additional information, such as a warm start point for each mode, to enable faster mixing across modes.
For this problem, we prove the first polynomial-time bound that works in a general setting, under a natural assumption that each component contains significant mass relative to the others when tilted towards the corresponding warm start point. For this, we introduce a modified version of the Annealed Leap-Point Sampler (ALPS).
Similarly to ALPS, we define distributions tilted towards a mixture centered at the warm start points, and
at the coldest level, use teleportation between warm start points to enable efficient mixing across modes. In contrast to ALPS, our method does not require Hessian information at the modes, but instead estimates component partition functions via Monte Carlo. This additional estimation step is critical in allowing the algorithm to handle target distributions with more complex geometries besides approximate Gaussian. For the proof, we show convergence results for Markov processes when only part of the stationary distribution is well-mixing and estimation for partition functions for individual components of a mixture. We numerically evaluate our algorithm's mixing performance on a mixture of heavy-tailed distributions, comparing it against the ALPS algorithm on the same distribution.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Submission Number: 19520