TrajP-L: A Trajectory-Based Plugin with LoRA for Sampling Direction Correction in Distilled Diffusion Models

Zongqing Li, Guangyi Wang, Qingxin Ye, Zhihui Liu, Ganjun Zheng, Jiashun Lin, Song-Zhi Su

Keywords: Diffusion Models, Distillation, Trajectory-based, Plugin

TL;DR: We propose TrajP-L for fast distillation—builds student model via LoRA, corrects sampling direction with trajectory coordinates, cuts discretization errors.

Abstract: Diffusion models (DMs) have shown remarkable capability in image synthesis, yet they typically require hundreds of sampling steps to produce high-quality outputs. To alleviate this inefficiency, prior work has explored distilling the sampling trajectories of pre-trained models. However, these approaches often disrupt the original parameter space and incur substantial distillation training costs. Recent findings suggest that the sampling space of DMs can be effectively captured by as few as three basis vectors, with the resulting low-dimensional trajectories exhibiting strong structural similarity. Based on this insight, we propose TrajP-L (Trajectory-Based Plugin with LoRA), a trajectory similarity-based learnable plugin. It achieves efficient DMs distillation via the synergy of LoRA and a trajectory correction module (TrajP). Specifically, we construct a student model by combining LoRA with the weights of a pre-trained model to initialize the distillation process. We then extract the coordinate information of the current and next sampling timesteps from a fitted 3D trajectory representation, and employ TrajP to refine the student’s sampling direction. Extensive experiments show that TrajP-L requires only a small number of sampling trajectories for fine-tuning, while substantially mitigating discretization errors. For example, on CIFAR-10, TrajP-L trains on merely 5k trajectories for 10 minutes on a single NVIDIA RTX 3090 GPU, improving DDIM performance from 169.50 FID (NFE=2) to 5.02.

Primary Area: generative models

Submission Number: 7605