Real-time video super-resolution using lightweight depthwise separable group convolutions with channel shuffling

Zhijiao Xiao, Zhikai Zhang, Kwok-Wai Hung, Simon Lui

Abstract: Highlights • This work proposes a new network architecture for video super-resolution. • Depthwise and pointwise group convolution for frame alignment and generation. • Frames alignment without explicit motion compensation to save computations. • Extensive ablation studies to verify the benefits of the proposed framework. • Outperform real-time SOTA in terms of calculations, running time and parameters. Abstract In recent years, convolutional neural networks (CNNs) have accelerated the developments of video super resolution (SR) for achieving higher image quality. However, the computational cost of existing CNN-based video super-resolution is too heavy for real-time applications. In this paper, we propose a new video super-resolution framework using lightweight frame alignment module and well-designed up-sampling module for real-time processing. Specifically, our framework, which is called as Lightweight Shuffle Video Super-Resolution Network (LSVSR), combines channel shuffling, depthwise convolution and pointwise group convolution to significantly reduce the computational burden during frame alignment and high-resolution frame reconstruction. On the public benchmark datasets, our proposed network outperforms the state-of-the-art lightweight video SR networks in terms of objective (PSNR and SSIM) and subjective evaluations, number of network parameters and floating-point operations. Our network can achieve real-time 540P to 2160P 4 × super-resolution for more than 60fps using desktop GPUs or mobile phones with neural processing unit.