Representation Alignment for Inverse Problems with Diffusion and Flow-Based Models

Loukas Sfountouris, Giannis Daras, Paris Giampouras

Keywords: Diffusion models, Flow-based Models, Inverse Problems, Representation Alignment, Representation learning

Abstract: Enforcing alignment between the internal representations of diffusion or flow-based generative models and those of pretrained self-supervised encoders has recently been shown to provide a powerful inductive bias, improving both convergence and sample quality. In this work, we extend this idea to inverse problems, where pretrained generative models are employed as priors. We propose applying representation alignment (REPA) between diffusion or flow-based models and a DINOv2 visual encoder, to guide the reconstruction process at inference time. Although ground-truth signals are unavailable in inverse problems, we empirically show that aligning model representations of approximate target features can substantially enhance reconstruction quality and perceptual realism. We integrate REPA into multiple state-of-the-art inverse problem solvers, and provide extensive experiments confirming that our method consistently improves reconstruction quality and realism.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 94