Video Deblurring via Spatiotemporal Pyramid Network and Adversarial Gradient Prior

Tao Wang, Xiaoqin Zhang, Runhua Jiang, Li Zhao, Huiling Chen, Wenhan Luo

Abstract: Highlights • We propose a spatiotemporal pyramid module as a new tool to model spatiotemporal dynamics within the video for the specific video deblurring task. • We introduce the gradient space of the image into the discriminator in GAN. With the goal of fooling the discriminator in the differential space, it is easier for the deblurring method to generate sharp videos. • The proposed methods achieve the state-of-the-art results by comparing with the existing methods on benchmark datasets. Abstract Video deblurring is to restore sharp frames from a blurry sequence. It is a challenging low-level vision task because the blur caused by camera shake, object motions and depth variations is heterogeneous in both spatial and temporal dimensions. Traditional methods usually work on a fixed spatiotemporal scale. However, the spatiotemporal scale of blurs in the video can vastly vary in the real-world situation. To address this challenge, we propose a Spatiotemporal Pyramid Network (SPN) to dynamically learn different spatiotemporal cues for video deblurring. Specifically, inside SPN, a spatiotemporal pyramid module is employed to effectively capture both spatial information and temporal information from the blurry sequence in a pyramid mode. An image reconstruction module constructs the sharp center frame through the obtained spatiotemporal information. Additionally, inspired by the statistical image prior and adversarial learning, we extend SPN and propose a Spatiotemporal Pyramid Generative Adversarial Network (SPGAN), which conducts adversarial discrimination in the gradient space. It helps the network produce more realistic sharp video frames. Experiments conducted on benchmarks demonstrate that the proposed methods achieve state-of-the-art results in terms of PSNR, SSIM and visual quality.