Exploring Source View Capability: Improve Generalizable 3D Reconstruction with Multi-view Context from Source Views
Keywords: Generalizable 3D Reconstruction, Novel View Synthesis, NeRF, 3DGS
TL;DR: We present a novel supervision method for generalizable 3D reconstruction, which explores the capability of source views for supervision, and significantly improves the performance for both NeRF and 3DGS based backbones.
Abstract: Recent generalizable 3D reconstruction methods have been facing challenges in constructing geometry-consistent 3D features.
This is primarily due to the source views conveying redundant information to the sampled 3D points that they do not observe, resulting in the samples struggling to distinguish the correct observations of them.
We attribute this issue to that canonical supervision methods focus solely on the rendered target view from a single viewpoint, overlooking source views that capture the scene from different perspectives.
With this insight, we pioneer a supervision method for source views, which can be applied alongside existing target view supervision in each iteration.
Specifically, we define the Learned Geometry of the Scene (LGS) as source-view depth distributions, which are derived from the weights of source views for each sampled 3D point.
To regularize the LGS to better model the real-world geometry, we introduce a novel unsupervised learning objective, which mitigates the optimization bias in existing objectives and ensures the LGS is more concentrated near the real-world geometry surface.
Regularizing the LGS effectively helps filter out irrelevant source views for each sampled 3D point, and thus noticeably improves the performance of backbones.
Mathematical proof is provided to validate the proposed objective, and extensive experiments demonstrate that our supervision method significantly improves both NeRF- and 3DGS-based backbones with negligible computation overhead.
Primary Area: applications to computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5541
Loading