Experts on Demand: Dynamic Routing for Personalized Diffusion Models
Abstract: Diffusion models have excelled in the realm of image generation, owing to their expansive parameter space. However, most users only exploit a fraction of the available capabilities for specialized image categories synthesis. These specific requirements for individual users are often persistently fixed over the long term, for example, a pet store pursues images of cats and dogs, which poses an efficiency challenge due to the computational complexity involved. In this paper, we introduce Mixture of Expert Diffusion Models (MoEDM), a personalized and efficient strategy for large-scale diffusion models specific to certain applications. By employing dynamic routing, MoEDM selectively activates only indispensable neurons, thereby optimizing runtime performance for specialized tasks while minimizing computational costs. Our MoEDM doubles the sampling speed without compromising efficacy across various applications. Moreover, MoEDM's modular design allows straightforward incorporation of state-of-the-art optimization methods such as DPM-Solver and Latent Diffusion. Empirical assessments, validated by FID score, KID score and human evaluation, confirm the advantages of MoEDM in terms of both efficiency and robustness.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Abhishek_Kumar1
Submission Number: 2109
Loading