Experts on Demand: Dynamic Routing for Personalized Diffusion Models

Juntu Zhao, Jialing Zhang, Chongxuan Li, Dequan Wang

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: generative model, personalization, diffusion models, dynamic models

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Diffusion models have excelled in the realm of image generation, owing to their expansive parameter space. However, this complexity introduces efficiency challenges. Most users only exploit a fraction of the available capabilities for specialized image categories. In this paper, we introduce Mixture of Expert Diffusion Models (MoEDM), a tailored and efficient strategy for large-scale diffusion models specific to certain applications. By employing dynamic routing, MoEDM selectively activates only indispensable neurons, thereby optimizing runtime performance for specialized tasks while minimizing computational costs. Our MoEDM doubles the sampling speed without compromising efficacy across various applications. Moreover, MoEDM's modular design allows straightforward incorporation of state-of-the-art optimization methods such as DPM-Solver and Latent Diffusion. Empirical assessments, validated by FID and KID scores, confirm the advantages of MoEDM in terms of both efficiency and robustness.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5410