The Missing U for Efficient Diffusion Models

Sergio Calvo Ordoñez, Chun-Wun Cheng, Jiahao Huang, Lipei Zhang, Guang Yang, Carola-Bibiane Schönlieb, Angelica I Aviles-Rivero

Abstract: Diffusion Probabilistic Models stand as a critical tool in generative modelling, enabling the generation of complex data distributions. This family of generative models yields record-breaking performance in tasks such as image synthesis, video generation, and molecule design. Despite their capabilities, their efficiency, especially in the reverse process, remains a challenge due to slow convergence rates and high computational costs. In this paper, we introduce an approach that leverages continuous dynamical systems to design a novel denoising network for diffusion models that is more parameter-efficient, exhibits faster convergence, and demonstrates increased noise robustness. Experimenting with Denoising Diffusion Probabilistic Models (DDPMs), our framework operates with approximately a quarter of the parameters, and $\sim$ 30\% of the Floating Point Operations (FLOPs) compared to standard U-Nets in DDPMs. Furthermore, our model is notably faster in inference than the baseline when measured in fair and equal conditions. We also provide a mathematical intuition as to why our proposed reverse process is faster as well as a mathematical discussion of the empirical tradeoffs in the denoising downstream task. Finally, we argue that our method is compatible with existing performance enhancement techniques, enabling further improvements in efficiency, quality, and speed.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

We express our gratitude to all reviewers and the Action Editor for their valuable time and insightful comments. Following their suggestions, we have implemented key changes to our manuscript.

1. First, we have enhanced the presentation of our model's applicability across various contexts within the introduction. This improvement aims to highlight our contributions more effectively.

2. Second, we have expanded our discussion on the novelty of our work in the conclusion section. These modifications have been made to better articulate the uniqueness and impact of our model.

Code: https://github.com/Sergio20f/cDDPM

Assigned Action Editor: ~Jakub_Mikolaj_Tomczak1

Submission Number: 1934