Efficient Video Compressed Sensing Reconstruction via Exploiting Spatial-Temporal Correlation With Measurement Constraint

Zhichao Wei, Chunling Yang, Yunyi Xuan

Abstract: Recent deep learning-based video compressed sensing (VCS) methods have achieved promising results but still suffer from numerous hyper-parameters and inflexibility. This paper proposes a novel network for VCS, named STM-Net, to fast recover high-quality video frames by optionally exploiting Spatial-Temporal information with a Measurement constraint. Combining the merits of adaptive sampling and adaptive shrinkage-thresholding, we first propose an improved ISTA-Net+ for framewise independent reconstruction, called Unfolding Adaptive Shrinkage-Thresholding Network (UAST-Net). To get further non-key frames reconstruction improvement, we develop a two-phase joint deep reconstruction, including an Occlusion-Aware Temporal Alignment to avoid irrelevant information compensation and a Multiple Frames Fusion with proposed Spatial-Temporal Feature Weighting (STFW) module to guide attractive content extraction and discriminative features generation. Besides, we develop a measurement loss to reduce the solution space to facilitate network optimization. Experimental results demonstrate the superiority of the proposed STM-Net over the existing methods.