Exploring Data-Free LoRA Transferability for Video Diffusion Models

Yuchen Wang, Wenliang Zhong, Lichen Bai, zikai zhou, Shitong Shao, Bojun Cheng, Shuo Chen, Shuo Yang, Zeke Xie

TL;DR: This paper analyzes LoRA transfer failure on distilled video diffusion models, attributing it to spectral interference in shared routing subspaces. It proposes Cluster-Aware Spectral Arbitration, a data-free framework for effective LoRA reuse.

Abstract: Video diffusion models leveraging step distillation or causal distillation have achieved remarkable performance. However, adapting existing LoRAs to these variants remains a critical challenge due to weight space mismatches. We observe that direct application leads to style degradation and structural collapse, yet the underlying mechanisms remain poorly understood. To fill this gap, we delve into the weight space and identify that the incompatibility stems from spectral interference within shared functional clusters defined over singular subspaces. Specifically, our analysis reveals that while both paradigms respect spectral rigidity, they establish conflicting routing pathways that clash through constructive overload or destructive cancellation. To address this issue, we propose Cluster-Aware Spectral Arbitration (CASA), a data-free framework that dynamically arbitrates between safeguarding the target's manifold and restoring LoRA alignment based on spectral density. Extensive experiments demonstrate that CASA effectively mitigates artifacts and revives LoRA functionality. Our code is available at https://github.com/Noahwangyuchen/CASA.

Lay Summary:

Video generation systems can create realistic videos, but they are expensive to run. To make them faster, researchers often build distilled versions of these models. At the same time, many users customize video models with LoRAs, which are small add-on modules that teach a model a new style, character, or visual behavior. A practical problem is that a LoRA trained for the original model often fails when reused on a faster distilled version, causing style loss or visual artifacts. Retraining LoRAs on each distilled model is also expensive and may require user data, raising privacy concerns.

We study why this happens by looking inside the model’s weights. Our analysis shows that distillation and LoRA usually keep the model’s overall weight structure stable, but they change how information flows through shared internal groups. When these changes interfere on important routes, generation can become unstable.

We propose CASA, a training-free method that transfers LoRAs by restoring useful style-related changes while protecting the routes needed by the distilled model. Experiments show that CASA improves both video quality and style preservation, making customized LoRAs easier to reuse on faster video generators.

Link To Code: https://github.com/Noahwangyuchen/CASA

Primary Area: Deep Learning->Generative Models and Autoencoders

Keywords: Video Diffusion Model, Low-Rank Adaptation, Transfer Learning

Submission Number: 10769