Spatial-Temporal Space Hand-in-Hand: Spatial-Temporal Video Super-Resolution via Cycle-Projected Mutual Learning

Mengshun Hu

Abstract:

Spatial-Temporal Video Super-Resolution (ST-VSR) aims
to generate super-resolved videos with higher resolution
(HR) and higher frame rate (HFR). Quite intuitively, pioneering two-stage based methods complete ST-VSR by
directly combining two sub-tasks: Spatial Video SuperResolution (S-VSR) and Temporal Video Super-Resolution
(T-VSR) but ignore the reciprocal relations among them.
Specifically, 1) T-VSR to S-VSR: temporal correlations help
accurate spatial detail representation with more clues; 2)
S-VSR to T-VSR: abundant spatial information contributes
to the refinement of temporal prediction. To this end, we
propose a one-stage based Cycle-projected Mutual learning network (CycMu-Net) for ST-VSR, which makes full
use of spatial-temporal correlations via the mutual learning between S-VSR and T-VSR. Specifically, we propose
to exploit the mutual information among them via iterative up-and-down projections, where the spatial and temporal features are fully fused and distilled, helping the
high-quality video reconstruction. Besides extensive experiments on benchmark datasets, we also compare our proposed CycMu-Net with S-VSR and T-VSR tasks, demonstrating that our method significantly outperforms state-of-theart methods.