LightMotion: A Light and Tuning-free Method for Simulating Camera Motion in Video Generation

Quanjian Song, ZhiHang Lin, Zhanpeng Zeng, Ziyue Zhang, Liujuan Cao, Rongrong Ji

Keywords: Tuning-free, Lightweight, Video Generation, Camera Simulation

TL;DR: A Light and Tuning-free Method for Simulating Camera Motion in Video Generation

Abstract: Existing camera-controlled video generation methods face computational bottlenecks, either due to significant fine-tuning overhead or heavy inference processes. In this paper, we proposes LightMotion, a light and tuning-free method for simulating camera motion in video generation. Operating in the latent space, it eliminates additional fine-tuning, inpainting, and depth estimation, making it more streamlined than existing methods. The endeavors of this paper comprise: (i) The latent space permutation operation simulates three basic camera motions: panning, zooming, and rotation, whose combinations cover almost all real-world movements. (ii) The latent space resampling strategy combines background-aware sampling with cross-frame alignment, accurately filling new perspectives while maintaining coherence across frames. (iii) Our analysis reveals that the tuning-free permutation and resampling will cause an SNR shift in latent space, leading to poor-quality generation. To address this, we propose the latent space correction scheme, which mitigates the shift and consequently improves video quality. Extensive experiments validate the superiority of LightMotion over other baselines.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 8474