Dual-Tap Computational Photography Image Sensor With Per-Pixel Pipelined Digital Memory for Intra-Frame Coded Multi-Exposure

Navid Sarhangnejad, Nikola Katic, Zhengfan Xia, Mian Wei, Nikita Gusev, Gairik Dutta, Rahul Gulve, Peter Zhi Xuan Li, Hui Feng Ke, Harel Haim, Manuel Moreno-García, Kiriakos N. Kutulakos, Roman Genov

Abstract: A coded-exposure-pixel image sensor for computational imaging applications is presented. Each frame exposure time is divided into N subframes. Within each subframe, each pixel sorts photo-generated charge into two charge taps depending on that pixel's 1-bit binary code. N global updates of arbitrary pixel-wise codes are implemented in each frame to enable N short global pixel-specific subexposures within one frame. To make these subexposures global, two latches per pixel are utilized in a pipelined fashion. The code for the next subframe is loaded into latch 1 in a row parallel fashion, while the code for the current subframe is being applied by latch 2 globally for photo-generated charge sorting during the current subexposure. A 280 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</sup> × 176 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V</sup> image sensor prototype with 11.2-μm pixel pitch has been fabricated in a 0.11-μm CMOS image sensor (CIS) technology. The image sensor has been demonstrated in two computational photography applications, each using only a single frame of a video: 1) computing both albedo (a measure of reflectivity) and 3-D depth maps by means of structured-light imaging and 2) computing surface normals (3-D orientations) map by means of photometric stereo imaging. These demonstrations experimentally validate some of the unique capabilities of this computational image sensor, such as accurate 3-D visual scene reconstruction using only one camera, while maintaining its native specifications: the full spatial resolution and the maximum frame rate.