Impact of In-Pixel Processing Circuit Non-idealities on Multi-object Tracking Accuracy for Autonomous Driving

Janak Sharda, Qiucheng Wu, Shiyu Chang, Shimeng Yu

Abstract: Deep learning algorithms are robust to a small amount of noise in the input image. Traditionally, image signal processors (ISP) are used with the CMOS image sensor (CIS) to enhance image quality which consume additional energy and latency. Here, we evaluate an ISP-Iess CIS architecture in the presence of noise and other in-pixel circuit non-idealities for the autonomous driving application. By integrating the in-pixel processing circuits to CIS, we filter out the redundant frames and only pass the critical bit information downstream to the backend processor. Such in-pixel processing does not allow ISP operations to be applied to the captured raw image. To reflect these limitations, we model and apply circuit non-idealities to the regenerated artificial raw images as an input to the QDTrack network for multi-object tracking. We evaluate the accuracy loss on the BDD100K dataset and examine its sensitivity on each of the image processing steps. We observe an overall accuracy drop of less than 1.2% in Identification Fl-score (IDFl) and 2.1 % in Multi-Object Tracking Accuracy (MOTA), suggesting that an ISP-Iess in-pixel processing circuit is feasible to reject 40% redundant frames directly on CIS.