Gradient-Free Analytical Fisher Information of Diffused Distributions
Keywords: diffusion models; Fisher Information; analytical formulation
TL;DR: We derive the analytical form of the Fisher information of diffused distribution and use this understanding to derive theory and expedite downstream tasks.
Abstract: Diffusion models (DMs) have demonstrated powerful distributional modeling capabilities by matching the first-order score of diffused distributions.
Recent advancements have explored incorporating the second-order Fisher information, defined as the negative Hessian of log-density, into various downstream tasks and theoretical analysis of DMs.
However, current practices often overlook the inherent structure of diffused distributions, accessing Fisher information via applying auto-differentiation to the learned score network.
This approach, while straightforward, leaves theoretical properties unexplored and is time-consuming.
In this paper, we derive the analytical formulation of Fisher information (AFI) by applying consecutive differentials to the diffused distributions.
As a result, AFI takes a gradient-free form of a weighted sum (or integral) of outer-products of the score and initial data.
Based on this formulation, we propose two algorithmic variants of AFI for distinct scenarios.
When evaluating the AFI’s trace, we introduce a parameterized network to learn the trace.
When AFI is applied as a linear operator, we present a training-free method that simplifies it into several inner-product calculations.
Furthermore, we provide theoretical guarantees for both algorithms regarding convergence analysis and approximation error bounds.
Additionally, we leverage AFI to establish the first general theorem for the optimal transport property of the diffusion ODE deduced map.
Experiments in likelihood evaluation and adjoint optimization demonstrate the superior accuracy and reduced time-cost of the proposed algorithms.
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 383
Loading