TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
Keywords: Visual Generation, Diffusion Model, LoRA
TL;DR: A new diffusion model fine-tuning paradigm.
Abstract: Diffusion models have driven the advancement of vision generation over the past years.
However,
it is often difficult to apply these large models in downstream tasks,
due to massive fine-tuning cost.
Recently,
Low-Rank Adaptation (LoRA) has been applied for efficient tuning of diffusion models.
Unfortunately,
the capabilities of LoRA-tuned diffusion models are limited,
since the same LoRA is used for different timesteps of the diffusion process.
To tackle this problem,
we introduce a general and concise TimeStep Master (TSM) paradigm with two key fine-tuning stages.
In the fostering stage (1-stage),
we apply different LoRAs to fine-tune the diffusion model at different timestep intervals.
This results in different TimeStep LoRA experts that can effectively capture different noise levels.
In the assembling stage (2-stage),
we design a novel asymmetrical mixture of TimeStep LoRA experts,
via core-context collaboration of experts at multi-scale intervals.
For each timestep,
we leverage TimeStep LoRA expert within the smallest interval as the core expert without gating,
and use experts within the bigger intervals as the context experts with time-dependent gating.
Consequently,
our TSM can effectively model the noise level via the expert in the finest interval,
and
adaptively integrate contexts from the experts of other scales,
boosting the versatility of diffusion models.
To show the effectiveness of our TSM paradigm,
we conduct extensive experiments on three typical and popular LoRA-related tasks of diffusion models,
including
domain adaptation,
post-pretraining,
and
model distillation.
Our TSM achieves the state-of-the-art results on all these tasks,
throughout various model structures (UNet, DiT and MM-DiT) and visual data modalities (Image, Video),
showing its remarkable generalization capacity.
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 744
Loading