There and Back Again: On the relation between noises, images, and their inversions in diffusion models

Lukasz Staniszewski, Lukasz Kucinski, Kamil Deja

Abstract: Diffusion Models achieve state-of-the-art performance in generating new samples but lack low-dimensional latent space that encodes the data into meaningful features. Inversion-based techniques try to solve this issue by reversing the denoising process and mapping images back to their approximated starting noise. In this work, we thoroughly analyze this procedure and focus on the relation between the initial Gaussian noise, the generated samples, and their corresponding latent encodings obtained through the DDIM inversion. First, we show that latents exhibit structural patterns in the form of less diverse noise predicted for smooth image regions. Next, we explain the origin of this phenomenon, demonstrating that, during the first inversion steps, the noise prediction error is much more significant for the plain areas than for the rest of the image. Finally, we present the consequences of the divergence between latents and noises by showing that the space of image inversions is notably less manipulative than the original Gaussian noise. This leads to a low diversity of generated interpolations or editions based on the DDIM inversion procedure and ill-defined latent-to-image mapping. Code is available at https://github.com/luk-st/taba.