SADA: Stability-guided Adaptive Diffusion Acceleration

Ting Jiang, Yixiao Wang, Hancheng Ye, Zishan Shao, Jingwei Sun, Jingyang Zhang, Zekai Chen, Jianyi Zhang, Yiran Chen, Hai Li

TL;DR: Training-free acceleration of generative models by adpatively exploit sparsity with principled approximation.

Abstract:

Diffusion models have achieved remarkable success in generative tasks but suffer from high computational costs due to their iterative sampling process and quadratic‐attention costs.
Existing training-free acceleration strategies that reduce per-step computation cost, while effectively reducing sampling time, demonstrate low faithfulness compared to the original baseline.
We hypothesize that this fidelity gap arises because (a) different prompts correspond to varying denoising trajectory, and (b) such methods do not consider the underlying ODE formulation and its numerical solution.
In this paper, we propose **Stability-guided Adaptive Diffusion Acceleration (SADA)**, a novel paradigm that unifies step-wise and token-wise sparsity decisions via a single stability criterion to accelerate sampling of ODE-based generative models (Diffusion and Flow-matching).
For (a), SADA adaptively allocates sparsity based on the sampling trajectory.
For (b), SADA introduces principled approximation schemes that leverage the precise gradient information from the numerical ODE solver.
Comprehensive evaluations on SD‐2, SDXL, and Flux using both EDM and DPM++ solvers reveal consistent $\ge 1.8\times$ speedups with minimal fidelity degradation (LPIPS $\leq 0.10$ and FID $\leq 4.5$) compared to unmodified baselines, significantly outperforming prior methods.
Moreover, SADA adapts seamlessly to other pipelines and modalities: It accelerates ControlNet without any modifications and speeds up MusicLDM by \(1.8\times\) with \(\sim 0.01\) spectrogram LPIPS.
Our code is available at: [https://github.com/Ting-Justin-Jiang/sada-icml](https://github.com/Ting-Justin-Jiang/sada-icml).

Lay Summary: (1) Generating from AI models requires many computational steps, which consume a lot of time and energy. (2) To tackle this challenge, we built SADA, a plug-in to accelerate these generative AI models without degrading their generation quality. (3) This can significantly improve the efficiency of professional workflows, and our method can be adapted to Image, Music, Audio, and potentially Video generation.

Link To Code: https://github.com/Ting-Justin-Jiang/sada-icml

Primary Area: Deep Learning->Generative Models and Autoencoders

Keywords: Diffusion Model, Efficient Algorithm, Training-Free Acceleration

Submission Number: 12564