Zero-shot Image Restoration via Diffusion Inversion

Hamadi Chihaoui, Abdelhak Lemkhenter, Paolo Favaro

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Inverse problems, Image restoration, Zero-shot learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Recently, various methods have been proposed to solve Image Restoration (IR)
tasks using a pre-trained diffusion models leading to state-of-the-art performance.
A common characteristic among these approaches is that they alter the diffusion
sampling process in order to satisfy the consistency with the corrupted input image.
However, this choice has recently been shown to be sub-optimal and may cause
the generated image to deviate from the data manifold. We propose to address this
limitation through a novel IR method that not only leverages the power of diffusion
but also guarantees that the sample generation path always lies on the data manifold.
One choice that satisfies this requirement is not to modify the reverse sampling ,
i.e., not to alter all the intermediate latents, once an initial noise is sampled. This
is ultimately equivalent to casting the IR task as an optimization problem in the
space of the diffusion input noise. To mitigate the substantial computational cost
associated with inverting a fully unrolled diffusion model, we leverage the inherent
capability of these models to skip ahead in the forward diffusion process using
arbitrary large time steps. We experimentally validate our method on several image
restoration tasks. Our method SHRED achieves state of the art results on multiple
zero-shot IR benchmarks especially in terms of image quality quantified using FID.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5248