DYNAMIC NOVEL VIEW SYNTHESIS FROM UNSYNCHRONIZED VIDEOS USING GLOBAL-LOCAL MOTION CONSISTENCY PRIOR

Xinzhuo Zhang, Junyu Zhu, Hao Zhu, Yaning Li, Qi Zhang, Zhan Ma, Xun Cao

Keywords: Dynamic novel view synthesis, Unsynchronized multi-view videos, Global-local motion consistency

TL;DR: We use a novel motion consistency prior to correct camera timing errors in dynamic novel view synthesis, significantly improving quality and accuracy.

Abstract: Dynamic novel view synthesis (D-NVS) critically depends on hardware-based synchronization. Current approaches that accommodate unsynchronized settings within the widely-used NeRF or GS frameworks often struggle with local minima, particularly in textureless scenes or when multi-view videos exhibit large misalignments. To tackle this issue, we propose a novel 3D global–2D local motion consistency prior, which evaluates the alignment between predicted scene flow projections and pre-computed optical flows across multi-view videos. Our analysis reveals that the motion, produced by the anisotropy of projected global scene flow across different views, is inherently more effective for correcting temporal misalignments compared to the near-isotropic appearance typically leveraged in NeRF or GS. Extensive experiments on public datasets demonstrate the versatility of our loss function across various D-NVS architectures (NeRF and GS), achieving a 50% reduction in synchronization errors and a PSNR improvement of up to 4dB, thereby outperforming existing state-of-the-art methods.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 5409