Accelerating Diffusion Models for Inverse Problems through Shortcut Sampling

Gongye Liu, Haoze Sun, Jiayi Li, Fei Yin, Yujiu Yang

Keywords: zero-shot, inverse problems, image restoration, diffusion models, super resolution, debluring, colorization

TL;DR: We propose a novel zero-shot framework for solving inverse problems. Compared to SOTA, it achieves competitive performance with fewer steps.

Abstract:

Recently, diffusion models have demonstrated a remarkable ability to solve inverse problems in an unsupervised manner. Existing methods mainly focus on modifying the posterior sampling process while neglecting the potential of the forward process.
In this work, we propose Shortcut Sampling for Diffusion (SSD), a novel pipeline for solving inverse problems. Instead of initiating from random noise, the key concept of SSD is to find the "Embryo", a transitional state that bridges the measurement image $y$ and the restored image $x$. By utilizing the "shortcut" path of "input-Embryo-output", SSD can achieve precise restoration with reduced steps.
To obtain the Embryo in the forward process, We propose Distortion Adaptive Inversion~(DA Inversion). Moreover, we apply back projection and attention injection as additional consistency constraints during the generation process.
Experimentally, we demonstrate the effectiveness of SSD on several representative IR tasks. Compared to state-of-the-art zero-shot methods, our method achieves competitive results with only 30 NFEs. Moreover, SSD with 100 NFEs can outperform state-of-the-art zero-shot methods in certain tasks.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5719