Time-Varying LoRA: Towards Effective Cross-Domain Fine-Tuning of Diffusion Models
Keywords: Low-rank adaptation, diffusion models, cross-domain learning
TL;DR: This paper introduces a novel time-varying low-rank adapter that offers an effective fine-tuning framework for domain flow generation, and generation-based cross-domain learning methods to mitigate domain shifts.
Abstract: Large-scale diffusion models are adept at generating high-fidelity images and facilitating image editing and interpolation. However, they have limitations when tasked with generating images in dynamic, evolving domains. In this paper, we introduce Terra, a novel Time-varying low-rank adapter that offers a fine-tuning framework specifically tailored for domain flow generation. The key innovation of Terra lies in its construction of a continuous parameter manifold through a time variable, with its expressive power analyzed theoretically. This framework not only enables interpolation of image content and style but also offers a generation-based approach to address the domain shift problems in unsupervised domain adaptation and domain generalization. Specifically, Terra transforms images from the source domain to the target domain and generates interpolated domains with various styles to bridge the gap between domains and enhance the model generalization, respectively. We conduct extensive experiments on various benchmark datasets, empirically demonstrate the effectiveness of Terra. Our source code is publicly available on https://github.com/zwebzone/terra.
Primary Area: Other (please use sparingly, only use the keyword field for more details)
Submission Number: 9139
Loading