SfPUEL: Shape from Polarization under Unknown Environment Light

Youwei Lyu, Heng Guo, Kailong Zhang, Si Li, Boxin Shi

Keywords: shape-from-polarization, photometric 3D reconstruction, physics-based vision

TL;DR: Single-view shape from polarization by integrating pretrained model knowledge under unknown environment illumination.

Abstract: Shape from polarization (SfP) benefits from advancements like polarization cameras for single-shot normal estimation, but its performance heavily relies on light conditions. This paper proposes SfPUEL, an end-to-end SfP method to jointly estimate surface normal and material under unknown environment light. To handle this challenging light condition, we design a transformer-based framework for enhancing the perception of global context features. We further propose to integrate photometric stereo (PS) priors from pretrained models to enrich extracted features for high-quality normal predictions. As metallic and dielectric materials exhibit different BRDFs, SfPUEL additionally predicts dielectric and metallic material segmentation to further boost performance. Experimental results on synthetic and our collected real-world dataset demonstrate that SfPUEL significantly outperforms existing SfP and single-shot normal estimation methods. The code and dataset is available at https://github.com/YouweiLyu/SfPUEL.

Primary Area: Machine vision

Submission Number: 6750