Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging

Bhargav Ghanekar, Salman Siddique Khan, Pranav Sharma, Shreyas Singh, Vivek Boominathan, Kaushik Mitra, Ashok Veeraraghavan

Abstract:

Passive, compact, single-shot 3D sensing is useful in
many application areas such as microscopy, medical imaging, surgical navigation, and autonomous driving where
form factor, time, and power constraints can exist. Obtaining RGB-D scene information over a short imaging distance, in an ultra-compact form factor, and in a passive,
snapshot manner is challenging. Dual-pixel (DP) sensors
are a potential solution to achieve the same. DP sensors
collect light rays from two different halves of the lens in
two interleaved pixel arrays, thus capturing two slightly
different views of the scene, like a stereo camera system.
However, imaging with a DP sensor implies that the defocus blur size is directly proportional to the disparity seen
between the views. This creates a trade-off between disparity estimation vs. deblurring accuracy. To improve this
trade-off effect, we propose CADS (Coded Aperture DualPixel Sensing), in which we use a coded aperture in the
imaging lens along with a DP sensor. In our approach,
we jointly learn an optimal coded pattern and the reconstruction algorithm in an end-to-end optimization setting.
Our resulting CADS imaging system demonstrates improvement of >1.5 dB PSNR in all-in-focus (AIF) estimates and
5-6% in depth estimation quality over naive DP sensing for
a wide range of aperture settings. Furthermore, we build
the proposed CADS prototypes for DSLR photography settings and in an endoscope and a dermoscope form factor.
Our novel coded dual-pixel sensing approach demonstrates
accurate RGB-D reconstruction results in simulations and
real-world experiments in a passive, snapshot, and compact
manner.