DomainStudio: Fine-Tuning Diffusion Models for Domain-Driven Image Generation using Limited Data

JingYuan Zhu, Huimin Ma, Jiansheng Chen, Jian Yuan

Keywords: Image generation, Diffusion models, Domain-driven, Limited data

TL;DR: This work explores few-shot and domain-driven image generation compatible with unconditional and text-to-image diffusion models.

Abstract: Denoising diffusion probabilistic models (DDPMs) have been proven capable of synthesizing high-quality images with remarkable diversity when trained on large amounts of data. Classic unconditional and modern large-scale conditional diffusion models are still vulnerable to overfitting when fine-tuned on extremely limited data. Existing works have explored subject-driven generation using a reference set containing a few images. However, few prior works explore DDPM-based domain-driven generation, which aims to learn the features of target domains while maintaining diversity. This paper proposes a novel DomainStudio approach to adapt DDPMs pre-trained on large-scale source datasets to target domains using limited data. It is designed to keep the diversity of subjects provided by source domains and get high-quality and diverse adapted samples in target domains. We propose to keep the relative distances between adapted samples to achieve considerable generation diversity. In addition, we further enhance the learning of high-frequency details for better generation quality. Our approach is compatible with both unconditional and text-to-image DDPMs. This work makes the first attempt to realize unconditional DDPM-based few-shot image generation, achieving better results than current state-of-the-art GAN-based approaches. It also significantly relieves overfitting for domain-driven text-to-image generation, expanding the applicable scenarios of modern large-scale text-to-image diffusion models.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 845