NeISF: Neural Incident Stokes Field for Geometry and Material Estimation

Chenhao Li, Taishi Ono, Takeshi Uemori, Hajime Mihara, Alexander Gatto, Hajime Nagahara, Yusuke Moriuchi

Abstract: Multi-view inverse rendering is the problem of estimating the scene parameters such as shapes, materials, or il-luminations from a sequence of images captured under dif-ferent viewpoints. Many approaches, however, assume single light bounce and thus fail to recover challenging sce-narios like inter-reflections. On the other hand, simply ex-tending those methods to consider multi-bounced light re-quires more assumptions to alleviate the ambiguity. To address this problem, we propose Neural Incident Stokes Fields (NeISF), a multi-view inverse rendering framework that reduces ambiguities using polarization cues. The pri-mary motivation for using polarization cues is that it is the accumulation of multi-bounced light, providing rich infor-mation about geometry and material. Based on this knowl-edge, the proposed incident Stokes field efficiently models the accumulated polarization effect with the aid of an orig-inal physically-based differentiable polarimetric renderer. Lastly, experimental results show that our method outper-forms the existing works in synthetic and real scenarios.