Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models

Published: 21 Sept 2023, Last Modified: 02 Nov 2023NeurIPS 2023 posterEveryoneRevisionsBibTeX
Keywords: Inverse Problems, Posterior Sampling, Latent Diffusion Model, Stable Diffusion, Sample Recovery
TL;DR: We present the first framework to solve inverse problems with a pre-trained latent diffusion model, which is the backbone of large-scale foundational generative models such as Stable Diffusion.
Abstract: We present the first framework to solve linear inverse problems leveraging pre-trained \textit{latent} diffusion models. Previously proposed algorithms (such as DPS and DDRM) only apply to \textit{pixel-space} diffusion models. We theoretically analyze our algorithm showing provable sample recovery in a linear model setting. The algorithmic insight obtained from our analysis extends to more general settings often considered in practice. Experimentally, we outperform previously proposed posterior sampling algorithms in a wide variety of problems including random inpainting, block inpainting, denoising, deblurring, destriping, and super-resolution.
Supplementary Material: zip
Submission Number: 9046
Loading