Monocular Dynamic View Synthesis: A Reality Check

Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, Angjoo Kanazawa

Keywords: dynamic view synthesis, novel-view synthesis, single-view 3D, dynamic 3D, 3D vision, NeRF

Abstract: We study the recent progress on dynamic view synthesis (DVS) from monocular video. Though existing approaches have demonstrated impressive results, we show a discrepancy between the practical capture process and the existing experimental protocols, which effectively leaks in multi-view signals during training. We define effective multi-view factors (EMFs) to quantify the amount of multi-view signal present in the input capture sequence based on the relative camera-scene motion. We introduce two new metrics: co-visibility masked image metrics and correspondence accuracy, which overcome the issue in existing protocols. We also propose a new iPhone dataset that includes more diverse real-life deformation sequences. Using our proposed experimental protocol, we show that the state-of-the-art approaches observe a 1-2 dB drop in masked PSNR in the absence of multi-view cues and 4-5 dB drop when modeling complex motion. Code and data can be found at http://hangg7.com/dycheck.

TL;DR: Existing works on dynamic view synthesis from monocular video actually evaluate on protocols that are essentially multi-view. We propose an actual monocular dataset and evaluation protocols that show there's much room for improvement.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2210.13445/code)