From compressive sampling to compressive tasking: retrieving semantics in compressed domain with low bandwidth

Zhihong Zhang, Bo Zhang, Xin Yuan, Siming Zheng

Abstract:

High-throughput imaging is highly desirable in intelligent analysis of computer vision
tasks. In conventional design, throughput is limited by the separation between physical
image capture and digital post processing. Computational imaging increases throughput
by mixing analog and digital processing through the image capture pipeline. Yet,
recent advances of computational imaging focus on the “compressive sampling”, this
precludes the wide applications in practical tasks. This paper presents a systematic
analysis of the next step for computational imaging built on snapshot compressive
imaging (SCI) and semantic computer vision (SCV) tasks, which have independently
emerged over the past decade as basic computational imaging platforms.
SCI is a physical layer process that maximizes information capacity per sample while
minimizing system size, power and cost. SCV is an abstraction layer process that analyzes
image data as objects and features, rather than simple pixel maps. In current practice,
SCI and SCV are independent and sequential. This concatenated pipeline results
in the following problems: i) a large amount of resources are spent on task-irrelevant
computation and transmission, ii) the sampling and design efciency of SCI is attenuated,
and iii) the fnal performance of SCV is limited by the reconstruction errors of SCI.
Bearing these concerns in mind, this paper takes one step further aiming to bridge the
gap between SCI and SCV to take full advantage of both approaches.
After reviewing the current status of SCI, we propose a novel joint framework by conducting
SCV on raw measurements captured by SCI to select the region of interest, and
then perform reconstruction on these regions to speed up processing time. We use our
recently built SCI prototype to verify the framework. Preliminary results are presented
and the prospects for a joint SCI and SCV regime are discussed. By conducting computer
vision tasks in the compressed domain, we envision that a new era of snapshot
compressive imaging with limited end-to-end bandwidth is coming.