Self-Supervised Video Super-Resolution by Spatial Constraint and Temporal Fusion

Cuixin Yang, Hongming Luo, Guangsen Liao, Zitao Lu, Fei Zhou, Guoping Qiu

Abstract: To avoid any fallacious assumption on the degeneration procedure in preparing training data, some self-similarity based super-resolution (SR) algorithms have been proposed to exploit the internal recurrence of patches without relying on external datasets. However, the network architectures of those “zero-shot” SR methods are often shallow. Otherwise they would suffer from the over-fitting problem due to the limited samples within a single image. This restricts the strong power of deep neural networks (DNNs). To relieve this problem, we propose a middle-layer feature loss to allow the network architecture to be deeper for handling the video super-resolution (VSR) task in a self-supervised way. Specifically, we constrain the middle-layer feature of VSR network to be as similar as that of the corresponding single image super-resolution (SISR) in a Spatial Module, then fuse the inter-frame information in a Temporal Fusion Module. Experimental results demonstrate that the proposed algorithm achieves significantly superior results on real-world data in comparison with some state-of-the-art methods.