Fourier Token Merging: Understanding and Capitalizing Frequency Domain for Efficient Image Generation

Jiesong Liu, Xipeng Shen

Keywords: Efficient transformers

Abstract:

Image generation requires intensive computations and faces challenges due to long latency.
Exploiting redundancy in the input images and intermediate representations throughout the neural network pipeline is an effective way to accelerate image generation.
Token merging (ToMe) exploits similarities among input tokens by clustering them and merges similar tokens into one, thus significantly reducing the number of tokens that are fed into the transformer block.
This work introduces Fourier Token Merging, a new method for understanding and capitalizing frequency domain for efficient image generation.
By introducing frequency token merging, we find that transforming the token into the frequency domain representation for clustering can better exert the ability of clustering based on the underlying redundancy after de-correlation.
Through analytical and empirical studies, we demonstrate the benefits of using Fourier clustering over the original time domain clustering.
We experimented fourier token merging on the stable diffusion model, and the results show up to 25\% reduction in latency without impairing image quality.
The code is available at https://github.com/Fred1031/Fourier-Token-Merging.

Primary Area: Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Submission Number: 24047