StereoCrafter-Zero: Zero-Shot Stereo Video Generation with Noisy Restart

Jian Shi, Qian Wang, Zhenyu Li, Ramzi Idoughi, Wenqing Cui, Peter Wonka

Keywords: Stereo Synthesis, Video Generation

Abstract:

Generating high-quality stereo videos requires consistent depth perception and temporal coherence across frames. Despite advances in image and video synthesis using diffusion models, producing high-quality stereo videos remains a challenging task due to the difficulty of maintaining consistent temporal and spatial coherence between left and right views.
We introduce \textit{StereoCrafter-Zero}, a novel framework for zero-shot stereo video generation that leverages video diffusion priors without requiring paired training data. Our key innovations include a noisy restart strategy to initialize stereo-aware latent representations and an iterative refinement process that progressively harmonizes the latent space, addressing issues like temporal flickering and view inconsistencies.
In addition, we propose the use of dissolved depth maps to streamline latent space operations by reducing high-frequency depth information.
Our comprehensive evaluations, including quantitative metrics and user studies, demonstrate that \textit{StereoCrafter-Zero} produces high-quality stereo videos with enhanced depth consistency and temporal smoothness. In terms of epipolar consistency, our method achieves an $11.7\%$ improvement in MEt3R score over the current state-of-the-art. Furthermore, user studies indicate strong perceptual gains over the previous arts, with an $8.0\%$ higher perceived frame quality and $10.9\%$ higher perceived temporal coherence.
Our code will be made publicly available upon acceptance of this manuscript.

Primary Area: generative models

Submission Number: 3739