Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention
Keywords: video super-resolution, adaptive, memory and computation efficiency
Abstract: Recently, Vision Transformer has achieved great success in recovering missing details in low-resolution sequences, i.e. the video super-resolution (VSR) task.
Despite its superiority VSR accuracy, the heavy computational burden as well as the large memory footprint hinders the deployment of Transformer-based VSR models on constrained devices, e.g. smart
phones and consumer electronic products.
In this paper, we address the above issue by proposing a novel feature-level masked processing framework: VSR with Masked Intra and inter frame Attention (MIA-VSR).
The core of MIA-VSR is leveraging feature-level temporal continuity between adjacent frames to reduce redundant computations and make more rational use of previously enhanced SR features.
Concretely, we propose an intra-frame and inter-frame attention block which takes the respective roles of past features and input
features into consideration and only exploits previously enhanced features to provide supplementary information.
In addition, an adaptive block-wise mask predicting module is developed to skip unimportant computations according to feature similarity between adjacent frames.
We conduct detailed ablation studies to validate our contributions and compare the proposed method with recent state-of-the-art VSR approaches.
The experimental results demonstrate that MIA-VSR improves
the memory and computation efficiency over state-of-the-art methods, without trading off PSNR accuracy.
Primary Area: representation learning for computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3
Loading