FreqCa: Accelerating Diffusion Models via Frequency Decomposition Caching

Jiacheng Liu, Peiliang Cai, Qinming Zhou, Yuqi Lin, Deyang Kong, Benhao Huang, Yupei Pan, Haowen Xu, Chang Zou, Junshu Tang, Shikang Zheng, Linfeng Zhang

Keywords: Diffusion Models, Image generation, Video generation, Model Acceleration, Feature Cache

TL;DR: FreqCa accelerates diffusion models by reusing low-frequency features and predicting high-frequency ones, slashing memory usage by 99% with a novel cumulative caching strategy.

Abstract:

The application of diffusion transformers is suffering from their significant inference costs. Recently, feature caching has been proposed to solve this problem by reusing features from previous timesteps, thereby skipping computation in future timesteps.
However, previous feature caching assumes that features in adjacent timesteps are similar or continuous, which does not always hold in all settings.
To investigate this, this paper begins with an analysis from the frequency domain, which reveal that \emph{different frequency bands in the features of diffusion models exhibit different dynamics across timesteps.} Concretely, low-frequency components, which decide the structure of images, exhibit higher \emph{similarity} but poor continuity. In contrast, the high-frequency bands, which decode the details of images, show significant continuity but poor similarity. These interesting observations motivate us to propose \textbf{Freq}uency-aware \textbf{Ca}ching (\textbf{FreqCa})
which directly reuses features of low-frequency components based on their similarity, while using a second-order Hermite interpolator to predict the volatile high-frequency ones based on its continuity.
Besides, we further propose to cache Cumulative Residual Feature (CRF) instead of the features in all the layers, which reduces the memory footprint of feature caching by \textbf{99\%}.
Extensive experiments on FLUX.1-dev, FLUX.1-Kontext-dev, Qwen-Image, and Qwen-Image-Edit demonstrate its effectiveness in both generation and editing. \emph{Codes are available in the supplementary materials and will be released on GitHub.}

Primary Area: generative models

Submission Number: 2446