Blind Inverse Problem Solving Made Easy by Text-to-Image Latent Diffusion

Michail Dontas, Yutong He, Naoki Murata, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov

Keywords: blind inverse problem solving, diffusion model, text-to-image generative model

TL;DR: We use text-to-image latent diffusion model to solve a broad range of blind inverse problem with no additional training

Abstract: This paper considers blind inverse image restoration, the task of predicting a target image from a degraded source when the degradation (i.e. the forward operator) is unknown. Existing solutions typically rely on restrictive assumptions such as operator linearity, curated training data or narrow image distributions limiting their practicality. We introduce LADiBI, a training-free method leveraging large-scale text-to-image diffusion to solve diverse blind inverse problems with minimal assumptions. Within a Bayesian framework, LADiBI uses text prompts to jointly encode priors for both target images and operators, unlocking unprecedented flexibility compared to existing methods. Additionally, we propose a novel diffusion posterior sampling algorithm that combines strategic operator initialization with iterative refinement of image and operator parameters, eliminating the need for highly constrained operator forms. Experiments show that LADiBI effectively handles both linear and challenging nonlinear image restoration problems across various image distributions, all without task-specific assumptions or retraining.

Submission Number: 46