Multiplane NeRF-Supervised Disentanglement of Depth and Camera Pose from Videos

Yang Fu, Ishan Misra, Xiaolong Wang

Keywords: Multiple Image Plane, Disentanglement from Video

Abstract: We propose to perform self-supervised disentanglement of depth and camera pose from large-scale videos. We introduce an Autoencoder-based method to reconstruct the input video frames for training, without using any ground-truth annotations of depth and camera. The encoders for our model will estimate the monocular depth and camera pose as the disentangled representations. The decoder will then construct a Multiplane NeRF representation based on the depth encoder feature, and perform rendering to reconstruct the input frames with the estimated camera. The disentanglement is learned with the reconstruction error, based on the assumption that the scene structure does not change in short periods of time in videos. Once the model is learned, it can be applied to multiple applications including depth estimation, camera pose estimation, and single image novel view synthesis. We show substantial improvements over previous self-supervised approaches on all tasks and even better results than counterparts trained with camera ground-truths in some applications. Our code will be made publicly available.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Unsupervised and Self-supervised learning

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2210.07181/code)