JOAStereo: Joint Learning Structured Light and Reconstruction for Generalizable Active Stereo

Yizhe Liu, Tong Jia, Xiao Yang, Xiaofang Li, Wenhao Li, Hao Wang, Dongyue Chen

Abstract: Active stereo is widely used in 3-D sensing for its advantages in cost and spatial resolution. Active stereo systems project hand-crafted patterns onto object surfaces to capture depth, which isolates structured light pattern design from scene information and reconstruction algorithms. Current active stereo introduces deep optics to reshape traditional design paradigms, which has achieved promising results. However, the performance in practice is limited due to the diffractive optical element (DOE) fabrication constraints and poor generalization. To address these problems, we propose a novel active stereo method called JOAStereo to jointly optimize the structured light pattern and the reconstruction network in an end-to-end manner. To this end, we design a digital speckle pattern (DSP) generation scheme and construct a differentiable active stereo model based on a projector according to geometric optics, which can be fabricated conveniently. In addition, to improve the generalization performance, we propose a feature decomposition mechanism based on a robust information bottleneck (RIB), which enables the network to extract domain-invariant representations (DIRs) without target domain-related annotations. Our method realizes the systematic matching between the optimized optical system and the reconstruction algorithm, which makes deep optics more practical. Extensive experiments demonstrate that our method can achieve state-of-the-art performance.