PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar

Tzofi Klinghoffer, Xiaoyu Xiang, Siddharth Somasundaram, Yuchen Fan, Christian Richardt, Ramesh Raskar, Rakesh Ranjan

Abstract:

3D reconstruction from a single-view is challenging because
of the ambiguity from monocular cues and lack of informa-
tion about occluded regions. Neural radiance fields (NeRF),
while popular for view synthesis and 3D reconstruction,
are typically reliant on multi-view images. Existing meth-
ods for single-view 3D reconstruction with NeRF rely on
either data priors to hallucinate views of occluded regions,
which may not be physically accurate, or shadows observed
by RGB cameras, which are difficult to detect in ambient
light and low albedo backgrounds. We propose using time-of-
flight data captured by a single-photon avalanche diode to
overcome these limitations. Our method models two-bounce
optical paths with NeRF, using lidar transient data for su-
pervision. By leveraging the advantages of both NeRF and
two-bounce light measured by lidar, we demonstrate that
we can reconstruct visible and occluded geometry without
data priors or reliance on controlled ambient lighting or
scene albedo. In addition, we demonstrate improved gen-
eralization under practical constraints on sensor spatial-
and temporal-resolution. We believe our method is a promis-
ing direction as single-photon lidars become ubiquitous on
consumer devices, such as phones, tablets, and headsets.