Efficiently Access Diffusion Fisher: Within the Outer Product Span Space

Fangyikang Wang, Hubery Yin, Shaobin Zhuang, Huminhao Zhu, Yinan Li, Lei Qian, Chao Zhang, Hanbin Zhao, Hui Qian, Chen Li

TL;DR: This paper reveals diffusion Fisher's outer-product structure, develops efficient algorithm, and verifies PF-ODE map's optimal transport.

Abstract:

Recent Diffusion models (DMs) advancements have explored incorporating the second-order diffusion Fisher information (DF), defined as the negative Hessian of log density, into various downstream tasks and theoretical analysis.
However, current practices typically approximate the diffusion Fisher by applying auto-differentiation to the learned score network. This black-box method, though straightforward, lacks any accuracy guarantee and is time-consuming.
In this paper, we show that the diffusion Fisher actually resides within a space spanned by the outer products of score and initial data.
Based on the outer-product structure, we develop two efficient approximation algorithms to access the trace and matrix-vector multiplication of DF, respectively.
These algorithms bypass the auto-differentiation operations with time-efficient vector-product calculations.
Furthermore, we establish the approximation error bounds for the proposed algorithms.
Experiments in likelihood evaluation and adjoint optimization demonstrate the superior accuracy and reduced computational cost of our proposed algorithms.
Additionally, based on the novel outer-product formulation of DF, we design the first numerical verification experiment for the optimal transport property of the general PF-ODE deduced map.

Lay Summary:

Diffusion models are a type of artificial intelligence that have been used for many tasks, like generating images. They work by gradually adding noise to data and then learning how to reverse this process to get back the original data. A key concept in these models is the diffusion Fisher information, which helps us understand how the data is spread out in the model.

Current methods for calculating this information are often inaccurate and slow. In our research, we discovered that the diffusion Fisher information is related to how the initial data and the model's 'guesses' (called scores) interact. Based on this, we developed two new, faster ways to estimate important aspects of the diffusion Fisher information. These new methods are more efficient because they avoid complex calculations and instead use simpler vector operations.

We also figured out how accurate these new methods are. Our experiments, such as when evaluating how well the model predicts likelihoods and in optimization tasks, showed that our new methods are not only more accurate but also save a lot of computing time. Additionally, our new understanding of the diffusion Fisher information allowed us to design the first experiment to numerically test an important property of a general map in the context of diffusion models. Overall, our work makes diffusion models more efficient and helps us better understand how they work.

Link To Code: https://github.com/zituitui/DiffusionFisher

Primary Area: Deep Learning->Generative Models and Autoencoders

Keywords: diffusion models; fisher information; optimal transport

Flagged For Ethics Review: true

Submission Number: 9641