Easing Training Process of Rectified Flow Models Via Lengthening Inter-Path Distance

Xu Shifeng, Yanzhu Liu, Adams Wai-Kin Kong

Keywords: Rectified Flow Models, Training, Easing, Distance-Aware Noise-Sample Matching, DANSM

TL;DR: easing the training process of rectified flow models by better noise-sample pairs in each training epoch

Abstract:

Recent research pinpoints that different diffusion methods and architectures
trained on the same dataset produce similar results for the same input noise.
This property suggests that they have some preferable noises for a given sample.
By visualizing the noise-sample pairs of rectified flow models and stable diffusion models in two-dimensional spaces,
we observe that the preferable paths, connecting preferable noises to the corresponding samples,
are better organized with significant fewer crossings comparing with
the random paths, connecting random noises to training samples.
In high-dimensional space, paths rarely intersect.
The path crossings in two-dimensional spaces indicate the shorter inter-path distance
in the corresponding high-dimensional spaces.
Inspired by this observation, we propose the Distance-Aware Noise-Sample Matching (DANSM) method
to lengthen the inter-path distance for speeding up the model training.
DANSM is derived from rectified flow models, which allow using a closed-form formula to calculate the inter-path distance.
To further simplify the optimization, we derive the relationship between inter-path distance and path length,
and use the latter in the optimization surrogate.
DANSM is evaluated on both image and latent spaces by rectified flow models and diffusion models.
The experimental results show that DANSM can significantly improve the training speed by 30\% $\sim$ 40\%
without sacrificing the generation quality.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2291